Guidance on

submissions

This document sets out the general framework for assessment

in the 2021 Research Excellence Framework (REF) and guidance

to UK higher education institutions about making submissions to

REF 2021. It includes guidance on procedures, the data that will

be required, and the criteria and definitions that will apply. The

deadline for submissions is midday, Friday 27 November 2020.

REF 2019/01

January 2019

Contents
 
Page
 
Executive summary  1
 
Purpose  1
 
Key points  1
 
Action required  2
 
Further information  2
 
Introduction  3
 
Part 1: Overview of the assessment framework  4 
Purpose  4
 
General principles  4
 
Framework for assessment  6
 
Publication of results and submissions  8
 
Equality and diversity  10
 
Timetable  12
 
Key changes since REF 2014  13 
Eligibility to participate in the REF for institutions  14
 
Part 2: Submissions  17
 
Scope of submissions  17
 
The submission process  21
 
Data verification  23
 
Interdisciplinary and collaborative research  24
 
Open access policy  26
 
Part 3: Data requirements and definitions  28 
Content of submissions  28
 
Part 3 Section 1: Staff details (REF1a/b)  28 
Part 3 Section 2: Research outputs (REF2)  48 
Part 3 Section 3: Impact (REF3)  68 
Part 3 Section 4: Environment data (REF4a/b/c)  77
 
Part 3 Section 5: Environment (REF5a/b)  80
 
Annex A: Assessment criteria and level definitions  84
 
Annex B: Quality profiles  87
 
Annex C: Definitions of research and impact for the REF  90
 
Annex D: Units of assessment  91
 
Annex E: Timetable  92
 
Annex F: Format and page limits for textual parts of submissions  93
 
Annex G: Impact case study template and guidance  95
 
Annex H: Institutional-level environment statement (REF5a)  99
 
Annex I: Unit-level environment template (REF5b)  100
 
Annex J: Standard data analyses  101 
Annex K: Output glossary and collection formats for REF2 and REF3  102 
Annex L: Reductions for staff circumstances  114
 
Annex M: Index of definitions and data requirements  117
 
Annex N: List of abbreviations  121

REF 2021

Guidance on submissions

Heads of higher education

institutions in the UK

Of interest to those

responsible for

Research

Reference

REF 2019/01

Publication date

January 2019

Enquiries from staff at

UK higher education

institutions

Email your institutional

REF contact. (These are

listed at www.ref.ac.uk

under Contact.)

Other enquiries

Hannah Daisley,

tel 0117 931 7486,

email info@ref.ac.uk

Executive summary

Purpose

This document:

•

sets out the framework and generic criteria for assessment

in the 2021 Research Excellence Framework (REF)

•

specifies the content, data requirements and related

definitions for submissions to REF 2021

•

guides higher education institutions (HEIs) on policy

and practical matters in preparing submissions.

Key points

The REF will be a process of expert review. Expert sub-panels

for each of 34 units of assessment (UOAs) will carry out the

assessment, working under the leadership and guidance of four

main panels.

In early 2020, the four UK higher education (HE) funding

bodies will invite UK HEIs to make submissions to REF 2021.

Each submission in each UOA will contain a common set of

data comprising:

Information on all staff in post with significant

responsibility for research on the census date, 31 July

2020; and information about former staff to whom

submitted outputs are attributed.

Details of assessable outputs produced in the submitted

unit during the publication period (1 January 2014 to

31 December 2020).

Case studies describing specific examples of impacts

achieved during the assessment period (1 August 2013 to

31 July 2020), underpinned by research in the period

1 January 2000 to 31 December 2020.

Data about research doctoral degrees awarded, research

income and income-in-kind related to the period 1 August

2013 to 31 July 2020.

An institutional-level environment statement, and a

completed template describing the submitted unit’s

research and impact environment, related to the period

1 August 2013 to 31 July 2020.

REF 2021

The deadline for submissions is 27 November 2020. Submissions will be assessed by

the REF panels during the course of 2021. Results will be published in December 2021, and

will be used by the HE funding bodies to inform research funding from the academic year

2022–23.

Action required

This document is for information and to guide institutions in preparing and collecting

data for inclusion in REF submissions. No action is required by HEIs at this stage.

Further information

Further information about the REF is available at www.ref.ac.uk.

Enquiries from members of staff at UK HEIs should be directed in the first instance to

their institutional REF contact. These contacts for each institution are listed at www.ref.ac.uk

under Contact.

Other enquiries should be addressed to info@ref.ac.uk.

REF 2021

Introduction

This document sets out the framework for assessment and administrative

arrangements for REF 2021. This guidance specifies the data requirements, definitions and

criteria that will apply, for submissions by HEIs

. It should be read in conjunction with REF

2019/02, the ‘Panel criteria and working methods’ (hereafter ‘Panel criteria’).

10.

The ‘Guidance on submissions’ and the ‘Panel criteria’ together describe

comprehensively the data required in submissions, and how panels will use the data in their

assessments. We may issue supplements to this guidance at later dates to clarify points of

detail regarding submissions, but such supplements will not request any new items of data.

11.

In autumn 2019, we will invite eligible HEIs to complete a survey to indicate whether

they intend to submit to REF 2021 and, if so, to provide details about their intended

submissions (see paragraphs 89 and 90).

12.

In early 2020, we will formally invite eligible HEIs to make submissions to REF 2021. We

will launch the submission system and provide accompanying technical guidance in early

2020. The deadline for submissions is 27 November 2020.

The four UK funding bodies have provided their definitions of an HEI for the purposes of the REF in

paragraph 52 of this document.

REF 2021

Part 1: Overview of the assessment framework

Purpose

13. The REF is the system for assessing research in UK HEIs. It was first conducted in 2014,

and replaced the previous Research Assessment Exercise (RAE).

14.

The REF is conducted jointly by Research England (RE), the Scottish Funding Council

(SFC), the Higher Education Funding Council for Wales (HEFCW) and the Department for

the Economy, Northern Ireland (DfE). The REF is managed by the REF team, based at RE,

on behalf of the four UK HE funding bodies, and is overseen by the REF Steering Group,

consisting of representatives of the four funding bodies. In this document, ‘we’ refers to

the REF team.

15.

The primary purpose of REF 2021 is to produce assessment outcomes for each

submission made by institutions. These outcomes deliver the wider threefold purpose of

the exercise, as follows:

The four HE funding bodies intend to use the assessment outcomes to inform the

selective allocation of their grant for research to the institutions which they fund,

with effect from 2022–23.

The assessment provides accountability for public investment in research and

produces evidence of the benefits of this investment.

The assessment outcomes provide benchmarking information and establish

reputational yardsticks, for use within the HE sector and for public information.

16.

In addition, the independent review of the REF, led by Lord Stern, identified three

further roles fulfilled by the REF:

•

to provide a rich evidence base to inform strategic decisions about national research

priorities

•

to create a strong performance incentive for HEIs and individual researchers

•

to inform decisions on resource allocation by individual HEIs and other bodies.

General principles

17.

The REF is a process of expert review. The recent independent review of the REF and

subsequent consultation confirmed widespread confidence in discipline-based expert

review, founded upon expert judgement.

18.

The REF is a single framework for assessment across all disciplines, with a common set

of data required in all submissions, standard definitions and procedures, and assessment

by expert panels against broad generic criteria, supplemented by specific criteria for each

main panel and its group of sub-panels where applicable. Expert panels will apply standards

of assessment consistently, working under the guidance of four main panels. Within this

single framework, differences in the nature of research across the disciplinary spectrum

may justify differences in the detailed approach to assessment. There is flexibility for panels

to develop specific aspects of the assessment criteria to ensure the assessment is sensitive

REF 2021

to these disciplinary differences. Panels have consulted with their subject communities and

with HEIs in doing so.

19.

REF 2021 has developed through an evolutionary and consultative process, building

on the successes of the previous REF, and earlier RAEs, and introducing key changes in

response to sector feedback. In implementing the recommendations of the Stern review,

informed by consultation feedback, the funding bodies have sought to strike a balance

between continuity and development, introducing changes where it is judged they can

bring demonstrable improvements, while recognising the efficiency gains in maintaining

continuity where possible.

20.

The following principles govern the conduct of the REF. They set the framework in which

the REF team co-ordinates the exercise and in which the four main panels and 34 sub-

panels will deploy their collective professional judgement to develop criteria, as set out in

the ‘Panel criteria’, for assessment and to assess submissions.

Equity:

All types of research and all forms of research output across all disciplines

shall be assessed on a fair and equal basis. Panels have been instructed to define

criteria and adopt assessment processes that enable them to recognise and

treat on an equal footing excellence in research across the spectrum of applied,

practice-based, basic and strategic research, wherever that research is conducted;

and for identifying excellence in different forms of research endeavour including

interdisciplinary and collaborative research, while attaching no greater weight to one

form over another. The REF aims to assess all types of research without distorting

the activity that it measures or encouraging or discouraging any particular type of

research activity, other than providing a general stimulus to enhancing the overall

achievements of the UK research base.

Equality:

HEIs are strongly encouraged to embed equality and diversity, and are

expected to comply with equality legislation in their processes for submitting staff

and outputs. We will require HEIs to develop, document and apply an internal code

of practice on their processes for identifying staff with significant responsibility for

research, for determining research independence, and for selecting outputs. To

support HEIs to adopt fair and transparent processes to represent the work of all

their researchers with significant responsibility for research, processes have been

developed to take account of staff circumstances which have affected an individual’s

or unit’s ability to work productively throughout the assessment period. The

measures to support equality and diversity within the REF are set out in paragraphs

44 to 48.

Transparency:

The credibility of the REF is reinforced by transparency in the process

through which decisions are made. The criteria and procedures that will be applied in

the assessment will be published in full, well in advance of institutions making their

submissions. The outcomes will be published in full and decision-making processes

at main and sub-panel levels will be explained openly. Paragraphs 34 to 43 set out

the information that will be published at the end of the exercise. We aim to make all

written documents and statements about the REF clear and consistent.

REF 2021

21.

The cost and burden of the REF should be the minimum possible to deliver a robust and

defensible process. The previous REF and RAEs have been highly cost-effective given the

value of public funds distributed through their outcomes (including the estimated cost to

HEIs). For example, we estimated the costs of REF 2014 to be less than one per cent of the

total public funding invested in research over a six-year period of an assessment cycle. We

will continue to weigh the burden on institutions against the need to ensure accountability

in disbursing public funds.

Framework for assessment

Units of assessment

22.

The REF will be a process of expert review, with discipline-based expert panels assessing

submissions made by HEIs in 34 UOAs. The UOAs are listed in Annex D. Detailed descriptors

of each UOA are set out in the ‘Panel criteria’, Part 2.

Submissions

23.

Institutions will make submissions by 27 November 2020. A submission comprises a

complete set of data about staff, outputs, impact and the environment returned by an HEI in

any of the 34 UOAs (see paragraph 65). Each submission in a UOA will contain, in summary:

REF1a/b:

Information on staff in post on the census date, 31 July 2020, with

significant responsibility for research; and information about former staff to whom

submitted research outputs are attributed.

REF2:

Details of assessable outputs that the submitted unit has produced during

the publication period (1 January 2014 to 31 December 2020). The total number of

outputs must equal 2.5 times the summed full-time equivalent (FTE) of the unit’s

submitted staff. Rounding to the nearest whole number will be applied to give a

whole number of outputs for submission

REF3:

Case studies describing specific examples of impacts achieved during the

assessment period (1 August 2013 to 31 July 2020) that are underpinned by excellent

research in the period 1 January 2000 to 31 December 2020.

REF4a/b/c:

Data about research doctoral degrees awarded, research income and

income-in-kind related to the period 1 August 2013 to 31 July 2020.

REF5a/b:

An institutional-level environment statement, and a completed template

describing the submitted unit’s research and impact environment, drawing on

quantitative indicators as appropriate, and related to the period 1 August 2013 to

31 July 2020.

REF6a/b:

information on staff circumstances, where a unit reduction or removal of

the requirement of the minimum of one output from each staff member is being

requested. This form must be completed by March 2020. Institutions will be able to

amend REF6a/b and make requests for further reductions at the point of submission.

2. Values ending in .5 should be rounded up.

REF 2021

Expert panels

24.

In each of the 34 UOAs, an expert sub-panel will conduct a detailed assessment of

submissions. The sub-panels will work under the leadership and guidance of four main

panels. The four main panels bear responsible for developing the panel criteria and working

methods, for ensuring adherence to the published procedures, for the consistent application

of the overall assessment standards, and for signing off the outcomes of the assessment.

25.

The expert panels were appointed by the four UK funding bodies through an open

process of nominations, as described in ‘Roles and recruitment of expert panels’ (REF

2017/03). Further information on the expert panels is set out in the ‘Panel criteria’, Part 1.

Assessment criteria

26.

Each sub-panel will examine the submissions made in its UOA, taking into account all

the evidence presented. They will use their professional judgement to form an overall view

about each submission. In doing so, the sub-panels will assess three distinct elements of

each submission, against the following generic criteria:

Outputs:

The sub-panels will assess the quality of submitted research outputs in

terms of their ‘originality, significance and rigour’, with reference to international

research quality standards. This element will carry a weighting of

60 per cent

in the

overall outcome awarded to each submission.

Impact:

The sub-panels will assess the ‘reach and significance’ of impacts on the

economy, society, culture, public policy or services, health, the environment or quality

of life that were underpinned by excellent research conducted in the submitted unit.

This element will carry a weighting of

25 per cent

Environment:

The sub-panels will assess the research environment of the submitted

unit in terms of its ‘vitality and sustainability’, including the approach to enabling

impact from its research, and its contribution to the vitality and sustainability of the

wider discipline or research base. This element will carry a weighting of

15 per cent

Panel criteria and working methods

27.

All sub-panels will apply the generic assessment criteria as set out in Annex A, and

will operate under a common assessment framework. They will assess submissions made

according to the standard definitions and common format set out in this document, and will

follow a common set of procedures and working methods in undertaking aspects of their

work (for example, in managing conflicts of interest).

28.

The ‘Panel criteria’ (REF 2019/02) includes details about how the sub-panels will apply

the generic assessment criteria, and sets out the working methods of the main and sub-

panels in undertaking their roles. It includes details of the generic criteria, working methods

and procedures to be followed by all panels, and includes details of any specific criteria to

be applied by each main or sub-panel.

29.

The main panels have developed a common set of criteria and working methods, setting

out any distinct criteria for its group of sub-panels where this is justified by differences in

the nature of research in the disciplines concerned. Guidance to the panels on developing

their criteria and working methods is published at www.ref.ac.uk, under Publications.

REF 2021

30.

The ‘Panel criteria’ includes further guidance to institutions about some particular forms

of evidence that would be appropriate to include in the textual parts of submissions. It

should therefore be read alongside this ‘Guidance on submissions’ publication: together the

documents set out comprehensively what information will be required in submissions, and

how the panels will assess the submissions.

Assessment outcomes

31.

For each submission, the sub-panels will develop a ‘sub-profile’ for each of the three

elements of the assessment (outputs, impact and environment). The sub-profiles will show

the proportions of activity (captured in REF 2-5) judged to meet each of four starred levels.

The starred levels are defined in Annex A.

32.

The three sub-profiles will be combined into an overall quality profile. An example

overall quality profile and the method for combining the sub-profiles is in Annex B. The

overall quality profile awarded to each submission will be the primary outcome of the REF,

to be published in December 2021.

33.

By presenting the outcomes in the form of quality profiles, we will ensure that pockets

of excellence are identified within the assessment outcomes.

Publication of results and submissions

34.

The primary outcome of the REF will be an overall quality profile for each submission,

and these will be published in December 2021. The quality profile will show the proportions

of research activity judged to meet each of four starred quality levels, in steps of one per

cent. Annex B describes how we will formulate and present quality profiles, including the

rounding methodology.

35.

Alongside the quality profile, the funding bodies will publish: the output, impact and

environment sub-profiles that were combined to produce the overall quality profile for each

submission; the FTE number of staff included in each submission (Category A submitted

staff); and the proportion of the eligible staff included as Category A submitted staff in

the submission. Paragraph 135 sets out the distinction between Category A eligible and

Category A submitted staff.

36.

Further reports and feedback from the exercise will be available early in 2022.

We expect the feedback to comprise:

A published report by each main panel confirming its working methods and providing

an overview of its observations about the state of research (strengths, weaknesses,

vitality of activity and scope of impacts achieved) in the areas falling within its remit.

These reports will include a section provided by each sub-panel.

Concise feedback on each submission summarising the reason for the quality

profile awarded, with reference to the published criteria of the sub-panel that

assessed it. We expect to send this feedback confidentially only to the head of the

institution concerned. In the case of joint submissions, we will provide this feedback

confidentially to the heads of all of the institutions involved.

REF 2021

Minutes of the main and sub-panel meetings for the assessment phase of the REF

will be published (redacted to maintain confidentiality), to provide a public record

of how the panels conducted their business.

A report by the REF director, detailing how the process was managed in

operational terms.

A report by the REF Equality and Diversity Advisory Panel, detailing its working

methods and observations about implementing the equality measures in the REF.

A report by the REF Interdisciplinary Research Advisory Panel, detailing its working

methods and observations about implementing measures to support the submission

and assessment of interdisciplinary research in the REF.

37.

The funding bodies will also publish analysis examining any equality impacts, including

an analysis of the number of outputs attributed to staff by protected characteristic, and

analysis over the REF period of staff contract change, including an analysis of any change by

protected characteristic.

38.

Those parts of submissions that contain factual data and textual information about

research activity will also be published and made available online in spring 2022.

Published information will include any research groups listed in a unit and textual

information, including impact case studies in which staff members may be referenced.

Staff members’ names and the associated Higher Education Statistics Agency (HESA) data

will not be published.

39.

We will separately list the details of submitted research outputs, and these will not

be listed by author name. This will indicate any research outputs that the HEI requested

be double-weighted and any research outputs that HEIs marked as ‘reserve’ outputs. The

published submissions will reflect the submissions made by HEIs, rather than the decisions

made by sub-panels, and no indication will be given of which requests for double-weighting

were accepted by the sub-panels, or which reserve outputs were assessed. We will publish

the submitted data on research doctoral degrees awarded and research income. We

will include submitted textual information about impact and the research environment.

Information on the attribution of outputs to individual staff members will not be published.

Any personal data contained in the outputs themselves, the impact case studies and

environment statements will not be removed before publication.

40.

Other than submitted impact case studies identified as ‘not for publication’, and those

elements within submitted case studies marked for ‘redaction’, the funding bodies intend to

publish submitted impact case studies as a searchable database, and intend to license the

content within this database under a Creative Commons Attribution 4.0 International licence

(CC-BY 4.0)

41.

The submission software will include a facility for HEIs to redact the listed outputs,

impact case studies or textual parts of submissions that should be omitted from the

Licence conditions available at https://creativecommons.org/licenses/by/4.0/legalcode.

10 REF 2021

published data for specific reasons, such as commercial sensitivity or security.

Further detailed guidance on this will be provided in advance of the release of the

submissions system.

42.

In reaching their judgements, panels will not take account of any information about

staff who are not submitted to the REF; also the published outcomes of the REF will not

include any information about non-submitted staff, other than as set out in paragraph 38.

43.

The results of the REF are not subject to appeal. The funding bodies have considered

carefully the question of appeals, and concluded that the absence of an appeals process

does not make the assessment process any less robust.

Equality and diversity

44.

The UK HE funding bodies are committed to supporting and promoting equality and

diversity in research careers, and strongly encourage transparency and fairness in decisions

made by HEIs to represent the excellent work of all their staff with significant responsibility

for research in submissions. Compliance with equality legislation is an obligation for

HEIs, and the four UK HE funding bodies have a statutory obligation as public bodies to

advance equality.

45.

The REF team is assisting the funding bodies in promoting equality and diversity in

research careers and in meeting their statutory obligations in a number of ways:

We will continue to work with the Equality and Diversity Advisory Panel (EDAP)

monitor and improve the representativeness of the appointed panels. This includes

reviewing the new requirements on nominating bodies to provide information on

how equality and diversity was taken into account when making nominations, ahead

of the next round of nominations; and analysing data on the protected characteristics

of the appointed panels and nominee pool.

We are ensuring that all main and sub-panels are briefed on all equality and relevant

employment legislation that will affect REF 2021. We have provided unconscious bias

training to the main and sub-panel chairs, and will deliver training across the full

panel membership in advance of the assessment phase.

We are strongly encouraging HEIs to fairly represent the research of all their excellent

researchers. Details are set out in paragraphs 156 to 200 on how institutions should

take into account the effect of individual circumstances on the ability of staff to work

productively during the assessment period. This is both in relation to a submitting

unit’s total output requirement and the requirement to submit a minimum of one

output for each Category A submitted staff member.

Each institution making a submission is required to develop, document and apply a

code of practice on their processes for identifying staff with significant responsibility

for research (where applicable), for determining research independence and for

The membership of EDAP is available at

www.ref.ac.uk

under ’Expert Panels’.

REF 2021 11

selecting outputs (see ‘Guidance on codes of practice’, REF 2019/03). Guidance on

drawing up a code of practice frames institutions’ decision-making processes in the

context of the principles of equality and all relevant legislation, thereby supporting

HEIs in meeting their own obligations.

Each institution will submit an institutional-level environment statement, providing

evidence about how equality and diversity in research careers is supported and

promoted across the institution.

Within each submission, as part of the description of the unit-level environment,

evidence will be required about how the submitted unit promotes equality and

diversity across its staff. The sub-panels will receive advice from EDAP to inform

their assessment of the ‘People’ section of the unit-level environment template.

As set out in paragraph 37, the funding bodies will monitor and analyse at sector

level any equality impacts, and will publish this analysis.

We will work with EDAP to develop appropriate briefing materials to support

institutions in taking account of equality and diversity when preparing submissions.

46.

These measures to promote equality and diversity through the REF have been informed

by consultation with the sector, and advice from EDAP, which was established specifically

to advise us on the development of these measures for the REF. The equality impact

assessment (EIA) undertaken for the policy development phase of the REF is available at

www.ref.ac.uk, under ‘Equality and Diversity’. The assessment will continue to be updated as

the exercise progresses.

47.

We will also continue to take expert advice on the implementation of these measures

throughout the REF from EDAP.

48.

We will also extract and pass to HESA the following data to enable verification: for

each individual returned in REF1a/b, their UOA and HESA staff identifier code (where

held) and date of birth. Hence, HEIs should ensure that decisions on staff submission

documented in line with their code of practice are consistent with relevant parts of their

annual individualised staff return to HESA. The 2018–19 and 2019–20 HESA individualised

staff return will require HEIs to return the REF UOA for all eligible academic staff, including

those identified as not having significant responsibility for research, to facilitate the funding

bodies’ equality analysis.

12 REF 2021

Timetable

49.

The timetable for REF 2021 is as follows, and is repeated in Annex E:

September 2017

Publication of ‘Initial decisions on the Research Excellence

Framework’ by the funding bodies, following consultation on

implementation of the Stern review recommendations (REF 2017/01)

October 2017

Publication of ‘Roles and recruitment of expert panels’ (REF 2017/03)

November 2017

Publication of ’Decisions on staff and outputs’ (2017/04)

March 2018

Panel membership for criteria phase announced

End of July 2018

Publication of draft ‘Guidance on submissions’ and ‘Panel criteria’

for consultation

15 October 2018

Close of consultation on draft ‘Guidance on submissions’ and

‘Panel criteria’

January 2019

Publication of final ‘Guidance on submissions’, ’Panel criteria’,

and ‘Guidance on codes of practice’; appointment of additional

EDAP members

Spring/

summer 2019

Institutions intending to make submissions to the REF submit their

codes of practice; invitation to request multiple submissions, case

studies requiring security clearance, and exceptions to submission

for small units (staggered deadlines in May, September and

December 2019); beta versions of the submission system will be

available in both test and live environments for institutions to use

Autumn 2019

Pilot of the REF submission system; survey of submissions

intentions opens; proposed date for inviting reduction requests

for staff circumstances

December 2019

Survey of submissions intentions complete; final deadline for

requests for multiple submissions, case studies requiring security

clearance, and exceptions to submission for small units; publication

of approved codes of practice

Early 2020

Formal release of the submission systems and accompanying

technical guidance; invitation to HEIs to make submissions; invitation

to nominate panel members and assessors for the assessment

phase; deadline for staff circumstances requests

Mid 2020

Appointment of additional members and assessors to panels

31 July 2020

Census date for staff; end of assessment period (for research

impacts, the research environment, and data about research income

and research doctoral degrees awarded)

27 November 2020

Closing date for submissions

31 December 2020

End of publication period (cut-off point for publication of research

outputs, and for outputs underpinning impact case studies)

29 January 2021

Deadline for providing further details for outputs pending

publication; redacted versions of impact case studies; and

corroborating evidence held for impact case studies

Throughout 2021

Panels assess submissions

December 2021

Publication of outcomes

Spring 2022

Publication of submissions, panel overview reports and sub-profiles

REF 2021 13

Key changes since REF 2014

50.

There are key differences between REF 2021 and REF 2014 concerning both the overall

assessment framework and the detailed data requirements and definitions. Therefore, HEIs

should not rely on their knowledge of the guidance and data requirements in previous RAEs,

and should refer to the guidance as set out in this document.

51.

We draw particular attention to the following changes:

HEIs will return all staff with significant responsibility for research, organised into

submitting unit(s) for return into the relevant UOA(s). The census date for staff will

be 31 July 2020.

The contribution of Category C staff (defined in paragraphs 374 to 376) should be

captured where applicable in the environment element of the exercise.

Each submitting unit will return a set number of outputs determined by the FTE of

Category A submitted staff. The total number of outputs must equal 2.5 times the

summed FTE of the unit’s submitted staff. This set number of outputs must comprise

of a minimum of one output attributed to each staff member returned, and no more

than five attributed to any staff member.

Institutions may return the outputs of staff previously employed as eligible where the

output was first made publicly available during the period of eligible employment,

within the set number of outputs required.

Outputs that meet the definition of an ‘in-scope’ output (defined in paragraph 223

to 224) must meet the open access requirements for REF 2021.

New measures have been introduced to ensure equality and diversity in research

careers will be promoted, including guidance and a template for codes of practice.

Institutions’ codes of practice should cover the processes for identifying staff with

significant responsibility for research (where applicable), for determining research

independence, and for selecting outputs.

Additional measures have been introduced to further support the submission and

assessment of interdisciplinary research.

Impacts on teaching within, as well as beyond, the submitting institution will be

eligible for submission. The unit’s approach to supporting and enabling impact

from research will be included within the environment template. We will require the

upfront submission of corroborating evidence for impact case studies.

Institutions will submit an institutional-level environment statement. This will inform

the panels’ assessment of the unit-level information, but will not be separately

scored. The information will be reviewed by a pilot panel to inform the inclusion of

an explicit institutional element in future exercises. The unit-level environment

template structure has been revised to incorporate new elements.

14 REF 2021

The weightings of the three elements have been revised to: outputs – 60 per cent;

impact – 25 per cent; environment – 15 per cent.

The sub-panels are being appointed in a two-stage process. Sub-panels with

sufficient expertise to contribute to the development of the criteria have been

appointed for 2018. These members will be joined by further sub-panel members

and assessors in 2020 to ensure the sub-panels have an appropriate breadth of

expertise and number of members necessary for the assessment phase.

There is greater consistency in the assessment process across the main panels.

The ‘Panel criteria’ has been developed with key aspects of the criteria combined

across the main panels, with variation where justified by discipline difference.

Eligibility to participate in the REF for institutions

52.

For the purposes of the REF, an HEI is defined as follows:

In Wales, the Further and Higher Education (1992) Act defines a Higher Education

Institution as ‘a university, an institution conducted by a higher education corporation

or a designated institution’ (see paragraphs 60 to 63).

In Northern Ireland, as defined in The Education and Libraries (NI) Order 1993,

‘a university, a college of education and any other institution which provides higher

education courses and is designated by regulations as a Higher Education Institution’.

In Scotland, the fundable bodies specified in Schedule 2 of the Further and Higher

Education (Scotland) Act 2005 (as amended), ‘as: (1) Institutions formerly eligible for

funding by the Scottish Higher Education Funding Council or as (2) Other institutions’.

In England, as set out in paragraphs 56 and 58.

53.

The funding bodies expect all HEIs (in all UK countries) submitting to the REF to have

research degree provision

. This includes institutions with validation, sub-contractual or

franchise arrangements with other HEIs. Institutions wishing to participate in the exercise

will also be required to meet the submission requirements that are set out in this

guidance document.

54.

Where an HEI does not have research degree provision, but is implementing a research

strategy that can demonstrate appropriate development and an increasing focus on

This is defi in accordance with the HESA defi of a research-based higher degree, as follows: A

research-based higher degree is a postgraduate programme comprising a research component (including

a requirement to produce original work) that is larger than any accompanying taught component when

measured by student eff The arrangements for assuring and maintaining the academic standards and

enhancing the quality of these programmes should be consistent with the expectations, practices and advice

and guidance of the ‘UK Quality Code for Higher Education’ https://www.qaa.ac.uk/quality-code. Some specialist

doctoral degrees, such as Doctor of Education (EdD) and Doctor of Clinical Psychology (DClinPsychol), may

include a research component but also include signifi taught components and supervised practice. These

degrees do not generally require the student to produce the same amount of original research as a PhD.

Students registered for a specialist doctoral degree should only be included as research students if they satisfy

both criteria.

REF 2021 15

research, the institution may contact the relevant funding body to request permission to

participate. Such institutions should contact the relevant funding body to discuss this matter

as soon as possible, and before 30 September 2019. Requests will be considered in relation

to the research profile of the institution, including any participation in REF 2014 or previous

RAEs, and its future research strategy.

Eligibility for HE providers in England

55.

In view of the English HE landscape changes, further guidance is provided in this section

relating to eligibility to participate in REF for different types of HE provider in England.

56.

The following institutions will be eligible to participate in the REF, where they also meet

the criteria in paragraph 53:

HEIs formerly eligible for HEFCE funding.

Alternative providers with university title.

57.

The Office for Students’ (OfS) Register of Providers will be fully implemented from

1 August 2019, as part of the Regulatory Framework for HE in England

. Eligibility to

participate in the REF will thereafter be based on the register, but will not supersede the

guidance in paragraph 56 for REF 2021. Therefore a provider meeting either or both of the

definitions set out in paragraphs 56 and 58, where they also meet the criteria in paragraph

52, will be eligible to participate.

58.

From 1 August 2019, the following providers will be eligible to participate in the REF,

where they also meet the criteria in paragraph 52:

Providers in the ‘approved (fee cap)’ category of the OfS register.

Providers in the ‘approved’ category of the OfS register, with university title.

59.

The Higher Education and Research Act (2017) specifies that only ‘eligible higher

education providers’ can receive funding from RE, including the quality-related research

funding informed by the outcomes of the REF. These eligible providers will be those in the

‘approved (fee cap)’ category of the OfS register. Those institutions meeting the definitions

in paragraph 56 only or meeting paragraph 58.b, will not be eligible to receive funding that

may be calculated with reference to the REF.

Eligibility for HE providers in Wales

60.

In view of changes to the Welsh HE regulatory landscape in recent years, further

guidance is provided in this section relating to eligibility to participate in the REF for different

types of HE providers in Wales.

61.

By way of clarification, reference to a designated institution in the context of the

definition of an HEI in Wales, provided in paragraph 52.a, relates to an institution designated

as eligible to receive funding support from HEFCW. This does not refer to an institution that

has courses designated for student support.

‘Securing student success: Regulatory framework for higher education in England’ (February 2018).

Available at https://www.officeforstudents.org.uk/

16 REF 2021

62.

In 2015 the Higher Education (Wales) Act 2015 gave HEFCW new regulatory powers. It

provided the opportunity for institutions in Wales that provided higher education and were

a charity the opportunity to be regulated by HEFCW through successfully applying for a fee

and access plan. Institutions that are regulated by HEFCW will not automatically become

designated to receive HEFCW funding.

63.

For avoidance of doubt, in Wales, only providers that are HEIs as defined at paragraph

52, have research degree provision and are in direct receipt of HEFCW funding in 2020/21

will be eligible to participate in the REF. Regulated institutions in Wales that do not satisfy

these three criteria will not be eligible to participate in the REF.

REF 2021 17

Part 2: Submissions

Scope of submissions

64.

Each HEI will submit all eligible staff it employs with significant responsibility for

research, organised into submitting unit(s) for return into the relevant UOA(s) as listed

in Annex D. In exceptional circumstances, an HEI may make a request to except from

submission a very small unit. The arrangements for this are set out in paragraphs 68 to 72.

An HEI will normally make only one submission in each UOA in which it submits, and only

exceptionally will this be waived to allow multiple submissions. Such exceptions are set out

in paragraph 73.

65.

A submission comprises a complete set of data about staff, outputs, impact and the

environment returned by an HEI in any of the 34 UOAs (as described in Part 3). A submission

provides evidence to the sub-panel about the activity and achievements of a ‘submitted

unit’. A submitted unit means the group or groups of staff identified by the HEI as working

primarily within the remit of a UOA and included in a submission, and by extension:

•

the research produced by the unit during the REF publication period

(1 January 2014 to 31 December 2020)

•

research related to that UOA and undertaken within the institution

(between 1 January 2000 and 31 December 2020), which underpins submitted

impact case studies

•

the structures and environment that support research and its application

or impact during the assessment period (1 August 2013 to 31 July 2020).

66.

Responsibility for mapping staff into submitted units lies with HEIs. A submitted unit

may, but need not, comprise only staff who work within a single department or other

organisational unit in the HEI. A submitted unit may alternatively comprise staff who work

in multiple organisational units in the HEI. The research carried out in a submitted unit must

relate primarily to the areas of research set out in the descriptor of the UOA in which it is

submitted. The UOA descriptors are set out in the ‘Panel criteria’, Part 2.

67.

We will cross-check the mapping of staff into UOAs against HESA cost centre data to

identify any significant anomalies. This measure is intended to underwrite the integrity of

the exercise. The funding bodies recognise that cost centre allocation is undertaken for

differing purposes to REF and do not intend that justification is routinely provided for HEIs’

mapping decisions. Where possible, we will cross-check this data, using the 2018-19 HESA

staff record, as part of the survey of submission intentions in 2019, and will follow up with

an HEI prior to the submission deadline where any significant anomalies are identified.

Requesting an exception from submission for small units

68.

Institutions will normally submit all eligible staff they employ with significant

responsibility for research, organised into submitting units as appropriate to the research

structures within that institution. They may exceptionally, and only with prior permission

from the REF director, request an exception from submission for very small units.

18 REF 2021

69.

Requests can be made for an exception from submission where the combined FTE of

staff employed with significant responsibility for research in the unit is lower than five FTE,

and where the research focus of these staff:

•

falls within the scope of one UOA

and

•

is clearly academically distinct from other submitting units in the institution

and

•

the environment for supporting research and enabling impact of each proposed

submitted unit is clearly separate and distinct from other submitting units in the

institution.

70.

The submitting institution will need to set out the case for an exception from submission

for the unit, which would normally fall under one of the following circumstances:

The research is in the scope of a UOA in which the institution has not previously

submitted, and has not been an area of investment and growth for the institution.

Where a previous REF submission has been made to this UOA, there has since

been a change in the staff profile in the research area in the institution.

71.

The REF director will decide on all such requests in consultation with chairs of the

relevant main and sub-panels. We will invite institutions to make any requests for

submission exceptions in early 2019. There will be three staggered deadlines for requests

in May, September and December 2019. If an HEI wishes to request an exception for more

than one unit, a separate application will be required for each request.

72.

Requests for submission exceptions are not binding. An institution may decide to return

the unit where they have been given approval for an exception.

Multiple submissions

73.

Institutions will normally make one submission in each UOA they submit in. They may

exceptionally, and only with prior permission from the REF director, make more than one

submission (multiple submissions) in the same UOA. These exceptions are:

Where an institution involved in a joint submission wishes to make an additional

individual submission in the same UOA.

Multiple submissions to Sub-panel 26 (Modern Languages and Linguistics) will

be permitted where one submission is in Celtic Studies and the other in Modern

Languages and Linguistics. This has been agreed in recognition of the special cultural

significance of Celtic Studies in parts of the UK, and the particular legal status of the

Welsh language in Wales.

Where HEIs merge after 1 July 2018, they may seek permission to make two separate

submissions in

all

of the UOAs in which they wish to submit, if for example they

anticipate difficulty in achieving academic cohesion between the merger date and the

submission date. Permission is unlikely to be granted to such HEIs to make separate

submissions only in some of the UOAs in which they wish to submit. In the event

that HEIs merged prior to 1 July 2018, the merged HEI should normally make

one

submission only to each UOA.

REF 2021 19

Where a sub-panel considers there is a case for multiple submissions in its UOA,

given the nature of the disciplines covered, the institution may request a multiple

submission. The following procedures apply:

i. The REF panels indicate in the ‘Panel criteria’ whether they expect to receive such

requests for multiple submissions, given the nature of the disciplines covered.

ii. The institution will need to make a convincing case that:

•

the bodies of research to be listed in each proposed submission fall within the

scope of the UOA but are clearly academically distinct from each other,

and

•

the environments for supporting research and enabling impact of each

proposed submitted unit are clearly separate and distinct, without significant

overlap in their research or staffing strategies, infrastructure, facilities or any

other aspects to be described in the textual parts of submissions.

iii.

The REF director will decide on all such requests in consultation with chairs of

the relevant main and sub-panels. In considering these requests, administrative

convenience of the submitting institution, or its preference for separate

assessment outcomes, will

not

be factors.

74.

Any HEI that wishes to make multiple submissions in relation to any of the exceptions

set out above will need to request prior permission from the REF director. We will invite

requests to make multiple submissions in early 2019. There will be three staggered

deadlines for requests in May, September and December 2019. Aside from the exception

mentioned in paragraph 73.c, a separate application to make multiple submissions will

be required for each UOA in which the HEI wishes to make multiple submissions. All

applications will be judged by the REF director, in consultation with the relevant main and

sub-panel chairs.

75.

Requests to make multiple submissions are not binding. An institution may decide

to make only one submission in a UOA where they have been given approval for multiple

submissions.

76.

Each submission will be awarded a single overall quality profile, made up of the three

quality sub-profiles for outputs, impact and environment. The outcomes will be provided at

this level for all UOAs. Additionally, in UOA 3 (Allied Health Professions, Dentistry, Nursing

and Pharmacy) and UOA 12 (Engineering), an institution may elect to receive multiple quality

sub-profiles for outputs (see ‘Panel criteria’, paragraphs 181 and 183).

77.

Where a single submission includes distinct organisational units or areas of research

and where the REF sub-panel considers it appropriate, the sub-panel will provide feedback

to the head of institution relating to the distinct units or areas of research. The main and

sub-panels will provide an overview of research within their discipline areas in the main

panel overview reports, including where appropriate at the level of any distinct areas of

research within the boundaries of one UOA.

20 REF 2021

Joint submissions

78.

We encourage joint submissions in a UOA by two or more UK institutions, where this

is the most appropriate way of describing research they have developed or undertaken

collaboratively. The method for joint submissions is described in paragraphs 79 to 84 and

is driven by two considerations:

Panels should receive joint submissions in the form of a unified entity, enabling them

to assess a joint submission in the same way as submissions from a single institution.

The REF team must be able to verify data in a joint submission through the HEIs to

which the data relates.

79.

Joint submissions may be made by HEIs that are from different countries in the UK.

In such cases any funding will be allocated to each HEI according to the funding method

of their respective funding body (see paragraph 83).

80.

Purely for REF administrative purposes, one HEI needs to be identified as the lead in

terms of management and data security of a joint submission. Two elements of the REF

data (REF3: Impact case studies; and REF5b: Environment template) will be submitted by

the lead HEI on behalf of all the other HEIs in the joint submission. Each HEI involved in the

joint submission will submit separate REF data in the following forms: REF1a/b (Staff details),

REF2 (Research outputs) and REF4a/b/c (Environment data). Each HEI’s institutional-level

statement (REF5a) will be provided to the sub-panel for the joint submission.

81.

In line with these submission arrangements, the submission system will include the

facility for HEIs involved in joint submissions to give ‘View’ and ‘Edit’ permissions to the

other HEIs involved in the relevant UOA. In order for panels to be able to judge the joint

submission like a single submission, the REF team will aggregate the data from each HEI

so that panels can receive and assess it as a coherent whole.

82.

The following rules apply:

Panels will assess the joint submission as they would a single submission, and the

outcome will be a single quality profile. The quality profile for a joint submission

will list the HEIs involved in alphabetical order, irrespective of which HEI took the

administrative lead in making the submission.

Panels will provide confidential feedback on joint submissions to the heads of all the

HEIs concerned; but the panels and the REF team will not comment specifically on the

contribution by an individual HEI to the overall quality profile.

In line with a general REF rule that no individual may be returned as Category A

submitted in more than one submission unless they hold a fractional employment

contract with more than one HEI (see paragraph 120.e), no individual can be returned

in a joint submission

and

in a submission from one HEI unless they hold two

separate employment contracts with two different HEIs.

REF 2021 21

Institutions involved in a joint submission that wish to make an additional individual

submission in the same UOA would normally be permitted to do so. Permission

should be sought through the normal multiple submissions request route as set out

in paragraphs 73 to 77.

Where an individual staff member has a contract of employment with more than

one partner institution in a joint submission, which makes them eligible for return,

that staff member should be returned in the joint submission by each employing

institution. The institutions must ensure that the total FTE value of the individual

sums to no more than the lower of 1.0 or the individual’s total contracted FTE duties.

Each HEI should attribute the same output(s) to the individual staff member in this

instance, with no more than five unique outputs in total attributed to them. The

information will be collated and presented to the sub-panel as a single submission.

Each distinct output attributed to the individual will be assessed only once and count

once towards the output sub-profile. Where the individual also has a contract of

employment with a further HEI, not included in the joint submission, all submitting

HEIs must follow the guidance for returning staff members who hold contracts across

multiple institutions in paragraph 120.e.

83.

Following the conclusion of the REF, each of the UK funding bodies will distribute

research funding in line with its individual funding formula. Assuming that, as in former

years, the FTE of staff submitted will be an element in those formulae, the funding bodies

envisage using the actual FTE of staff submitted by each HEI involved in a joint submission,

unless the HEIs involved propose a different, agreed percentage split of funding at the

time of submission. HEIs may also request that this percentage split is published in the

final results.

84.

Further guidance on the technical procedure for making joint submissions will

accompany the technical guidance on how to use the submissions system.

The submission process

Method of submission

85.

We are developing software for REF 2021 to collect submissions from HEIs, on behalf

of the four funding bodies. It will be a web-based application, with the application and data

storage hosted by a major cloud provider in one of their UK-based data centres, and will be

the only way HEIs can make a submission to the REF. A pilot version of the system will be

made available to HEIs in autumn 2019. The final version will be available in January 2020.

86.

The software development will be undertaken following an agile approach that should

allow the release of updates as the development progresses and the provision of access for

HEIs to beta versions of the software from early 2019 for both test and live environments.

While the intention is that any information entered into the submission system during 2019

would not be lost when updates are applied, where this is not possible then we will give

advance notice of what information may be lost during the update process.

22 REF 2021

87.

The submission system will allow HEIs to enter data by direct on-screen data entry or by

importing data in various file formats. Details of the file formats supported will be available

at www.ref.ac.uk, including the schemas for XML and JSON and templates for other file

formats. All data will be encrypted at rest and in transit between the servers and clients.

Access to the submission system will not be permitted to anyone other than authorised

personnel within each HEI, RE’s system administrators, and agencies involved in auditing

legal compliance such as for intellectual property rights.

Access to research outputs

88.

All outputs that are listed in submissions will be made available to the relevant

sub-panel, as follows:

For journal articles and conference proceedings, the submission must include a

Digital Object Identifier (DOI) wherever available, to enable the REF team to source

these outputs from the publishers. Where we cannot source particular outputs

from the publishers, the HEI must provide an electronic copy. Where supplementary

information has been published for the output, the submission must include a

DOI (or other URL, if no DOI is available) for this, where it differs to the DOI of the

submitted output.

For all other types of output, the HEI must provide an electronic copy wherever this is

available; or where it is not, a physical output or appropriate evidence of the output

(see Annex K).

Survey of submission intentions

89.

In autumn 2019, we will undertake a survey of HEIs about their submission intentions,

to assist with planning of the assessment phase. This will inform further appointments of

members and assessors to the sub-panels. Through the survey we will ask HEIs to indicate:

•

the likely volume and UOAs of staff with significant responsibility for research,

as detailed in the HESA staff return for 2018–19

•

the main areas of research and impact to be included in the submission and the

likely volume of work in each research area (this will need to be in sufficient detail

for panels to understand the breadth and depth of expertise required for the

assessment, and in particular to inform the recruitment of sub-panel members

and assessors)

•

the likely volume of work to be submitted in languages other than English.

90.

All institutions intending to participate in the REF will need to confirm this via the survey,

and complete the required information. Responses to the survey will not be binding, but

institutions should note that the information will be very helpful to the REF team and panels

in planning their work and especially to underpin the appointment of further full sub-panel

members, and output and impact assessors. Responses will be required by December 2019.

REF 2021 23

Data verification

91.

All information provided by HEIs in submissions to REF 2021 must be capable of

verification. We will check a proportion of submitted information from each institution as

a matter of course. Panel members will also be asked to draw attention to any data that

they would like us to verify, and this data will be investigated. HEIs should therefore be able

to provide justification for all information submitted.

92.

Following the REF 2014 process, for REF4 we will again align the data on research

doctoral degrees awarded and research income with HESA returns, and data on research

income-in-kind with information held by the Research Councils and the health research

funding bodies (as listed in paragraph 347). As described in Part 3, Section 4, the REF

submission system will limit the extent to which such data submitted by an HEI can exceed

their prior returns to HESA. While we recognise that the basis of returns for REF and other

data sets may differ and exact matches might not be possible, we will investigate instances

where there appear to be significant differences between submitted data and other returns.

93.

Institutions are encouraged to ensure HESA returns are as accurate as possible. We

will not schedule a formal period for adjusting HESA data being used to check information

submitted in REF4.

94.

For REF3, each submitted impact case study should include details of external sources

of information that could corroborate claims made about the impact of the submitted

unit’s research. As described in Part 3, Section 3, institutions are required to provide to the

REF team the corroborating evidence held for submitted impact case studies by 29 January

2021. This will be held by the REF team and will not be routinely provided to the sub-panels.

It will be made available to panels via panel-instigated audit. We will also audit a proportion

of case studies and will examine these sources during that process.

95.

The detailed arrangements for data verification and audit in REF 2021 are currently

being developed. We will publish the details of these arrangements in summer 2019.

96.

Where an HEI is unable to provide justification for any piece of information contained

in its submission, that information will be excluded from assessment. The funding bodies

will consider what further action to take in any case where serious discrepancies are found.

97.

We are conscious of the potential additional workload verification may cause HEIs, and

aim to minimise this. To this end, data checking and verification will normally be conducted

by correspondence with the REF team. If HEIs wish to make it available, our verification

team may also seek to rely on any relevant internal audit work that an HEI has undertaken

in preparing or submitting REF data. Members of the REF team and other officers of the

funding bodies reserve the right to visit HEIs to verify submission information.

Data protection

98.

We will collect, store and process all personal data submitted by HEIs to the REF in

accordance with current data protection legislation – the General Data Protection Regulation

(EU) 2016/679 and the Data Protection Act 2018. Information will be processed for the

purposes of conducting and evaluating the REF. Personal data will be shared via a secure

system with panel chairs, members, assessors, secretaries and observers, who are all bound

24 REF 2021

by confidentiality arrangements. It will also be shared with the four UK funding bodies that

need it to inform the selective distribution of public funds for research and to carry out their

statutory functions connected with funding HE:

•

Department for the Economy, Northern Ireland

•

Higher Education Funding Council for Wales

•

Research England

•

Scottish Funding Council.

99.

The organisations listed above will use the personal data to analyse and monitor REF

2021. This may result in personal data being released to other users including academic

researchers or consultants (commissioned by the funding bodies), to carry out research

or analysis, in accordance with the Data Protection Act 2018. Where personal data not

previously published is released to third parties, this will be anonymised where practicable.

As stated in paragraph 48, we will extract and pass some personal data to HESA to enable

data verification. We will also publish parts of submissions on the internet (as described

in paragraphs 39 to 40). Anonymised parts of the staff data will be passed to EDAP for the

purpose of monitoring the diversity of staff to whom differing numbers of outputs are

attributed in the exercise. HEIs should ensure that individuals whose work is included in

their submissions are aware of these uses, including the publication of submissions. We

will publish a full Privacy Notice that will provide further information on the collection,

processing and retention of personal data submitted to the REF.

100.

All organisations processing personally identifiable information should ensure

that information about its use is provided to the data subjects in accordance with the

transparency requirements of data protection law. To assist institutions in ensuring that

individuals included in their submissions are aware of these uses, we will provide a model

privacy notice for REF 2021, which institutions can adapt to their own circumstances.

Interdisciplinary and collaborative research

101.

An underpinning principle of the REF is that all types of research and all forms of

research output across all disciplines shall be assessed on a fair and equal basis, including

interdisciplinary and collaborative research. There have been concerns that the assessment

of interdisciplinary research has presented challenges in previous REF and RAE exercises,

due to the discipline-based structure of the UOAs, or that collaborative research has not

been encouraged due to the competitive nature of the exercise.

102.

A number of enhancements to the procedures for assessing interdisciplinary research

were introduced in REF 2014, including: broader UOAs with sub-panels that have the

expertise to assess a wider range of research, and the appointment of additional assessors

to extend the depth and breadth of panels’ expertise to undertake the assessment;

improved procedures for cross-referral, including the option to cross-refer individual

outputs for advice; and the facility for HEIs to identify those outputs which it considered to

be interdisciplinary, to draw this to the panels’ attention.

REF 2021 25

103.

The REF will support the equitable assessment of interdisciplinary research through

the following measures:

We have appointed an Interdisciplinary Research Advisory Panel (IDAP) to advise

the REF team, REF panel chairs and the UK funding bodies on the development

and implementation of measures to support the submission and assessment

of interdisciplinary research in the REF. IDAP will not advise on individual

assessments, but will advise on process, and provide advice and support for

cross-panel collaboration.

We have appointed at least one member to each main panel with specific

responsibility for providing guidance on the assessment of interdisciplinary

research, and who will join IDAP for the assessment phase. We will also appoint at

least two members of each sub-panel to the role of interdisciplinary research

(IDR) adviser to provide guidance on the assessment of interdisciplinary research

submitted in that UOA and to work with advisers in other sub-panels to ensure its

equitable assessment.

Working with IDAP, we have developed a definition of interdisciplinary research

for the REF to better enable HEIs to identify interdisciplinary outputs at the

point of submission (paragraph 273). The interdisciplinary identifier will allow

panels, working with their IDR advisers, to consider the most appropriate means

of assessing the output, including within the sub-panel, with advice from the

IDR advisers’ network, or with advice from another sub-panel. The use of the

interdisciplinary identifier is distinct from the cross-referral process

. The ‘Panel

criteria’ sets out further details on these processes. The IDR identifier will also

enable the funding bodies to conduct post-exercise analysis of interdisciplinary

research in REF 2021, enabling them to assess the effectiveness of the additional

measures put in place.

Working with IDAP, we have developed additional guidance for the sub-panels on

assessing interdisciplinary research outputs according to the generic assessment

criteria for outputs. This guidance is included in the ‘Panel criteria’, paragraph 196.

Informed by the survey of submission intentions, we will seek to appoint assessors

to work with more than one sub-panel, where there are strong cross-disciplinary

connections between particular sub-panels.

Within the institutional-level environment statement and the unit-level environment

template, HEIs can provide information about their approach to supporting

interdisciplinary research. The panels will give due credit where these arrangements

have enhanced the vitality and sustainability of the research environment.

A briefing document drawing together all guidance and criteria relating to

interdisciplinary research will be published in February 2019.

In addition to cross-referring parts of a submission between sub-panels for advice to inform the

assessment, parts of submissions may be made available to other sub-panels for the purposes of

calibration exercises. Calibration procedures are described in more detail in the ‘Panel criteria’, Part 5.

26 REF 2021

104.

The REF will support collaborative research through the following arrangements:

We encourage joint submissions in a UOA by two or more UK institutions, where

this is the most appropriate way of describing the research they have developed

or undertaken collaboratively. (See paragraphs 78 to 84.)

Outputs that are co-authored or co-produced and are listed in more than one

submission (whether within the same HEI or from different HEIs) may be listed in

any or all of those submissions, and will be assessed on an equal footing to any

other output.

Where a submitted impact is underpinned by collaborative research, each

submitting unit whose research made a distinct and material contribution to the

impact may submit that impact.

Within the environment template, a submitting unit should provide information

about how they support collaboration both within and beyond academia, and

panels will give due credit where these arrangements have enhanced the vitality

and sustainability of the research environment or the submitted unit’s contribution

to the wider research base, economy or society.

Open access policy

105.

The guidance on open access set out in this document supersedes the previously

published open access policy, circular letter(s) regarding the policy, and the FAQs. This

section sets out the policy intent. The detailed definitions and requirements are set out

in paragraphs 223 to 255.

106.

The four UK HE funding bodies believe that the outputs of publicly funded research

should be freely accessible and widely available. Open access research brings benefits

to researchers, students, institutions, governments, public bodies, professionals and

practitioners, citizen scientists and many others. Open access has the potential to make

research more efficient and impactful. In view of these benefits, and to embed open access

as an intrinsic part of the research process, the funding bodies have introduced a policy

requirement on open access in REF 2021.

107.

The intent of the REF open access policy is to provide a set of minimum requirements

for open access, while encouraging an environment where researchers and HEIs move

beyond the minimum requirements. HEIs can demonstrate where they have gone beyond

the requirements in the environment template (REF5b) in the research and impact strategy

section (see the ‘Panel criteria’, paragraph 346). The funding bodies encourage institutions

to take a proportionate view of the costs and benefits of making other types of outputs

(including monographs) available as open access.

108.

The open access policy applies to journal articles and conference contributions (with an

International Standard Serial Number (ISSN)) which are accepted for publication from 1 April

2016 and published on or before 31 December 2020. It requires these research outputs

to be made open access for those outputs to be eligible for submission in REF 2021. The

REF 2021 27

outputs should be deposited, discoverable, and free to read, download and search within,

by anyone with an internet connection. The funding bodies recommend that institutions

fully consider the extent to which authors currently retain or transfer the copyright of works

published by their researchers, as part of creating a healthy research environment.

109.

Authors and institutions can meet the policy requirement without necessarily incurring

any additional open access publication costs (such a through payment of an article

processing charge).

110.

Evidence gathered in 2017 indicates good progress is being made by the sector in

implementing the policy, and a range of systems and tools are being developed to assist

authors and institutions in making their outputs open. However, the funding bodies

recognise that the current structures and software solutions are still at an early stage,

and that it will take time to fully establish open access as an intrinsic part of the research

process. The funding bodies expect the sector and service providers to continue the

momentum to develop new tools to implement the policy, particularly relating to the

deposit requirements. In view of this, there are measures and exceptions which have been

developed to provide a degree of tolerance of non-compliance.

111.

The funding bodies recognise that information on deposit permissions, licences and

embargoes can sometimes be unclear, complex, or hard to find. Until significant progress

has been made to address this issue (including developing machine-readable licences and

permissions), it is reasonable for the sector to rely on shared services, including those

offered by SHERPA (Securing a Hybrid Environment for Research Preservation and Access).

Authors and institutions should feel comfortable acting on the information provided

by SHERPA in meeting REF 2021 open access requirements, and should not undertake

additional work to verify this information.

112.

Institutions are not expected to correspond with previous institutions to evidence that

outputs published while a staff member was previously employed elsewhere fulfilled the

requirements of the policy. This is the case even if the new employer intends to submit the

output to REF 2021. For example, when a researcher moves from the institution where the

output was published (X), to another institution (Y), the REF does not expect that institution

Y corresponds with institution X to seek and retain evidence of the output’s compliance.

Where an institution is unable to ascertain themselves if an output is compliant with the

policy, a policy exception can be applied (see paragraph 254.a).

113.

The intent of this policy is for the output to be made freely available. The policy

encourages outputs which are submitted with a deposit, technical, or other exception to

be made open access as soon as possible. However, this is not a requirement of the policy.

114.

The REF 2021 audit process will seek assurance that the information and data

submitted regarding compliance are accurate and reliable

Audit for the REF 2021 open access policy will be developed as part of the wider REF audit and data

verification arrangements (see paragraphs

91 to 97).

28 REF 2021

Part 3: Data requirements and definitions

Content of submissions

115.

Each submission will contain the core data outlined in sub-paragraphs a. to f., with

further details about each set out in this part of the document. (The REF numbering refers

to the name of the forms in which the data will be collected.)

Staff details (REF1a/b):

Information on Category A submitted staff in post on the

census date (31 July 2020) (REF1a); and information about former staff to whom

submitted outputs are attributed (REF1b).

Research outputs (REF2): Details of assessable outputs which the submitted unit

has produced during the publication period (1 January 2014 to 31 December 2020).

The total number of outputs must equal 2.5 times the summed FTE of the unit’s

submitted staff.

Impact case studies (REF3): Case studies describing specifi examples of impacts

achieved during the assessment period (1 August 2013 to 31 July 2020), underpinned

by excellent research in the period 1 January 2000 to 31 December 2020.

Environment data (REF4a/b/c): Data about research doctoral degrees awarded

(REF4a), research income (REF4b) and research income-in-kind (REF4c) related to

the period 1 August 2013 to 31 July 2020.

Environment (REF5a/b): An institutional-level environment statement (REF5a),

and a completed template describing the submitted unit’s research and impact

environment (REF5b), drawing on quantitative indicators as appropriate, and

related to the period 1 August 2013 to 31 July 2020.

REF6a/b:

information on staff circumstances, where a unit reduction or removal of

the requirement of the minimum of one output is being requested. This form must

be completed by March 2020. Institutions will be able to amend REF6a/b and make

requests for further reductions at the point of submission.

Part 3 Section 1: Staff details (REF1a/b)

116.

Each HEI participating in REF 2021 must return all eligible staff with significant

responsibility for research. Such staff will normally be identified by the core eligibility

criteria, as set out for ‘Category A eligible staff’ (see paragraph 117); however, where these

criteria will also identify staff who do not have significant responsibility for research, the

institution may develop and apply an agreed process to identify who among their staff

meeting the core eligibility criteria have significant responsibility for research. This process

must be documented in a code of practice (see ‘Guidance on codes of practice’, REF

2019/03). A diagram illustrating staff eligibility is set out in Figure 1.

REF 2021 29

Category A eligible staff

117.

Category A eligible staff are defined as academic staff with a contract of employment

of 0.2 FTE or greater, on the payroll of the submitting institution on the census date,

whose primary employment function is to undertake either ‘research only’ or ‘teaching and

research’

. Staff should have a substantive research connection with the submitting unit

(see paragraphs 123 to 127). Staff on ‘research only’ contracts should meet the definition

of an independent researcher (paragraphs 128 to 134).

118.

The funding bodies are aware that there are a very small number of instances where

Category A eligible staff are not returned to HESA due to internal employment structures.

They are working with the affected institutions, including the Universities of Oxford and

Cambridge in England and the University of the Highlands and Islands in Scotland, to ensure

that a robust alternative is in place to ensure that eligible and submitted staff are identified

in a fair and consistent manner.

119.

The funding bodies recognise that staff on ‘teaching and research’ contracts cannot

always be assumed to be independent researchers. Where this is the case, staff who are not

independent researchers should be identified as part of the process for identifying staff with

significant responsibility for research (see paragraphs 138 to 143).

120.

Regardless of their job title, all staff who satisfy the definition in paragraph 117, are

considered Category A eligible staff. By way of further guidance, please note that so long

as they satisfy the criteria at paragraph 117:

Staff who hold institutional/NHS joint appointments

are

considered Category A

eligible. For clinical academics where the HEI is the primary employer (and the

other contract is honorary) the staff member should be returned with the full FTE

of the primary employment contract with the HEI. Where a clinical academic holds

two employer contracts (for example, A+B contracts) they may be returned by the

HEI for that fraction of their employment with the HEI.

Pensioned staff who continue in salaried employment contracted to carry out

research and meet the defi in paragraph 117

are

considered Category A

eligible staff

Where academic staff are on unpaid leave of absence, or on secondment to an

organisation other than a UK HEI (as defined in the footnote to paragraph 9), on the

census date

and

are contracted to return to normal duties up to two years from the

start of their period of absence or secondment,

either

the seconded staff member

any staff recruited to cover their duties that meet the eligibility criteria should be

considered Category A eligible. The FTE of the post should be included only once

in the submission, and the minimum of one output required for whichever staff

member is returned. Submitting units may include the outputs of both staff in the

submitted output pool. Where these are attributable to the staff member who is

not returned, the staff details in REF1b should be completed.

Individuals whose primary employment function is to undertake either ‘research only’ or ‘teaching and

research’ are staff returned to the Higher Education Statistics Agency Staff Collection with an academic

employment function of either ‘Academic contract that is research only’ or ‘Academic contract that is both

teaching and research’ (identified as codes ‘2’ or ‘3’ in the ACEMPFUN field).

30 REF 2021

Where a staff member is working on secondment as contracted academic staff

at another UK HEI on the census date and meets the definition of Category A

eligible in both HEIs, the two institutions concerned should agree how the FTE is

to be apportioned to each, and the minimum of one output requirement must

be met by both institutions (this could be the same or a different output for each

HEI). The staff member’s total FTE may not exceed their contracted FTE with their

main employer. The FTE of any staff directly recruited to cover the secondment

at the ‘home’ HEI, who meet the definition of Category A eligible staff, will also be

considered eligible for return. The combined FTE of the seconded and cover staff

should not exceed the total contracted FTE of the post, and must not exceed 1.0

FTE in total. The minimum of one output requirement should also be met for the

cover staff.

Other than individuals on secondment on the terms described in sub-paragraph d.,

an individual will

only

meet the definition Category A eligible by more than one

HEI if they have a contract with and receive a salary from more than one HEI.

In such cases:

i. The two HEIs must ensure that the total FTE value of the individual sums to no

more than the individual’s total contracted FTE duties, and must not exceed 1.0

FTE in total. If any individual is returned in submissions with a contracted FTE

that sums to more than 1.0, the REF team will rectify this through verification,

and will apportion the FTE to each HEI pro-rata to the individual’s contracted FTE

at each HEI.

ii. The same research outputs may, but need not be, attributed to the staff

member in each submission.

No individual may be returned in more than one submission, except as described

in sub-paragraphs d. and e.:

Where an individual holds a joint appointment across two or more submitting

units within the same institution, the HEI must decide on one submission in

which to return the individual.

ii.

Where a staff member is serving a notice period at an institution, having started

employment with a different institution on the census date, if they meet the

eligibility criteria, the individual will be deemed eligible for return by the first

institution only.

Staff whose salary is calculated on an hourly or daily basis are eligible

only

if they

meet the definition in paragraph 117

and

on the census date have a contract

of employment of at least 0.2 FTE per year, over the length of their contract.

Institutions should calculate the mean FTE of these staff using the number of

hours or days worked in the HESA reporting years that fall wholly within the REF

assessment period (2014–15 to 2019–20), based on the standard hours or days of

a full-time employee at that institution.

REF 2021 31

Staff who hold more than one contract for different functions within the HEI, are

eligible if one of those contracts satisfies the definition of Category A eligible staff

in paragraph 117. Such staff should be returned with an FTE that is no greater than

that of the qualifying contract.

Where an individual holds one contract with multiple functions, the individual

should be returned with the FTE of the contract that makes them eligible for

submission to the REF, not the FTE specifically related to their research duties

within that contract.

121.

Academic staff who are employed by the submitting HEI and based in a discrete

department or unit outside the UK are eligible

only

if the primary focus of their research

activity on the census date is clearly and directly connected to the submitting unit based in

the UK. In assessing this, HEIs should be guided by the indicators suggested for evidencing

a substantive connection (see paragraphs 123 to 125).

122.

Staff described in paragraph 121 should be returned to HESA. No additional

information should be submitted; however, HEIs will need to be able to verify the

connection in the event of audit.

Substantive connection

123.

For staff employed on minimum fractional contracts (0.20 to 0.29 FTE) on the census

date, the HEI will need to provide a short statement (up to 200 words) evidencing the clear

connection of the staff member with the submitting unit. A range of indicators is likely to

evidence a substantive connection, including but not limited to:

•

evidence of participation in and contribution to the unit’s research environment,

such as involvement in research centres or clusters, research leadership activities,

supervision of research staff or supervision of postgraduate research (PGR) students

•

evidence of wider involvement in the institution, for example through teaching,

knowledge exchange, administrative, and/or governance roles and responsibilities

•

evidence of research activity focused in the institution (such as through publication

affi shared grant applications or grants held with the HEI)

•

period of time with the institution (including prospective time, as indicated through

length of contract).

124.

Staff who do not have a substantive research connection with the submitting unit will

not be eligible for inclusion, such as those who hold substantive research posts at another

institution (either within or outside the UK)

and

whose research is not clearly connected

with the submitted unit. A statement evidencing a substantive connection will not be

required for staff members with a contract of employment greater than or equal to 0.3

FTE on the census date; however, a substantive research connection remains an eligibility

requirement for all staff and HEIs will need to be able to verify this in the event of audit.

125.

Staff whose connection cannot be demonstrated to the satisfaction of the REF director,

as advised by the relevant sub-panel, will be considered ineligible and removed from the

REF database (see also paragraph 137).

32 REF 2021

126.

The funding bodies recognise that there are also particular personal and discipline-

related circumstances where the minimum fractional contract will commonly apply for staff

members who have a substantive connection with the submitting unit. Therefore, in these

instances, a statement evidencing a substantive connection will not be required for staff

with contract of employment between 0.20 and 0.29 FTE. These instances are as follows:

•

where the staff member has caring responsibilities

•

where the staff member has other personal circumstances (e.g. ill health, disability)

•

where the staff member has reduced their working hours on the approach

to retirement

•

where the fractional appointment refl normal discipline practice (for example,

where joint appointments with industry or practice are typical in the submitted unit).

127.

Institutions will need to identify the applicable circumstances in lieu of providing a

statement at the point of submission. This information will not be made available to panels.

No additional information should be submitted; however, HEIs will need to be able to verify

the circumstances in the event of audit. Where audit determines the cited circumstances are

not applicable, the funding bodies will seek assurance from the HEI that the staff member(s)

has a substantive connection as set out in paragraph 123.

Independent researchers

128.

Staff employed on ‘research only’ contracts must be independent researchers (defined

in paragraphs 131 to 133) to meet the definition of Category A eligible. All staff on ‘research

only’ contracts who are independent researchers will have significant responsibility for

research so should be returned as Category A submitted staff.

129.

Research assistants (sometimes also described as postdoctoral research assistants,

research associates or assistant researchers) as defined in paragraph 130, are not eligible

to be returned to the REF unless, exceptionally, they meet the definition of an independent

researcher (defined in paragraphs 131 to 133) on the census date

and

satisfy the definition

of Category A eligible staff in paragraph 117. They must not be listed as Category A

submitted staff purely on the basis that they are named on one or more research outputs.

130.

Research assistants are defined as academic staff whose primary employment

function is ‘research only’, and they are employed to carry out another individual’s research

programme rather than as independent researchers in their own right (except in the

circumstances described in paragraph 129). They are usually funded from research grants

or contracts from Research Councils, charities, the European Union (EU) or other overseas

sources, industry, or other commercial enterprises, but they may also be funded from the

institution’s own funds.

131.

For the purposes of the REF, an independent researcher is defined as an individual

who undertakes self-directed research, rather than carrying out another individual’s

research programme.

132.

Possible indicators of independence are listed below. Institutions should note that each

indicator may not individually demonstrate independence and where appropriate multiple

factors may need to be considered. The main panels have set out in the ‘Panel criteria’

REF 2021 33

(paragraphs 187 to 189) the indicators they consider appropriate for their disciplines.

The following indicators are considered appropriate by all main panels

•

leading or acting as principal investigator or equivalent on an externally funded

research project

•

holding an independently won, competitively awarded fellowship where research

independence is a requirement. An illustrative, but not exhaustive, list of

independent fellowships can be found at www.ref.ac.uk, under Guidance

•

leading a research group or a substantial or specialised work package.

133.

A member of staff is not deemed to have undertaken independent research purely

on the basis that they are named on one or more research outputs.

134.

Institutions are required to develop processes for determining research independence

in accordance with the guidance in paragraphs 128 to 133 and document these processes in

their code of practice (see REF 2019/03).

Category A submitted staff

135.

Category A submitted staff are defined as Category A eligible staff who have been

identified as having significant responsibility for research on the census date. This will

include all staff on ‘research only’ contracts who are ‘Category A eligible’. Staff on ‘teaching

and research’ contracts will be included according to one of the following approaches:

Where the ‘Category A eligible’ staff definition accurately identifies staff in the

submitting unit with significant responsibility for research, the unit should submit

100 per cent of staff.

Where the ‘Category A eligible’ staff defi does not accurately identify staff

(on ‘teaching and research’ contracts) in the submitting unit who have signifi

responsibility for research and are independent researchers, the institution will need

to implement processes to determine this. Institutions will need to develop, consult

with staff on, and document in their code of practice, the processes to be followed

for identifying who among those meeting the defi of ‘Category A eligible’ staff

have signifi responsibility, and are therefore in scope for submission. The onus

will be on institutions to provide evidence that ‘Category A eligible’ staff on ‘teaching

and research’ contracts who are not submitted do not have signifi responsibility

for research. Further guidance on this approach is set out in paragraphs 138 to 143.

136.

Where a Category A submitted staff member is deemed ineligible through audit (for

example, if they are not considered to meet the definition of an independent researcher, or

there is a dispute over which HEI employs the staff member) they will be removed from the

submission, and their FTE subtracted from the submitted total.

137.

Where a staff member is removed through audit, all of the outputs attributed to that

staff member in the submission will also be removed. Where this leads to a lower number

of outputs in the submission than is required for the new total submitted FTE, any ‘missing’

outputs will receive an unclassified score. Where the total number of outputs is higher than

the recalculated requirement, all remaining outputs will be included in the assessment.

34 REF 2021

Significant responsibility for research

138.

The funding bodies require institutions to submit all eligible staff with significant

responsibility for research. Staff with significant responsibility for research are those for

whom explicit time and resources are made available to engage actively in independent

research, and that is an expectation of their job role. The definition of research for the REF

is provided in Annex C.

139.

In many institutions and in many discipline areas the core eligibility criteria as set out

in paragraph 117 will accurately identify staff with significant responsibility for research.

For institutions where this is not uniformly the case and the approach set out in paragraph

135.b is followed, the following additional guidance is provided to support institutions to

identify staff with significant responsibility for research in a consistent way. In all instances

this should be based upon the expectations of staff as a function of employment, and not

upon the quality or volume of what has been delivered as a result of that employment

function. In developing their processes, it is expected that HEIs will adhere to the principles

underpinning the policy, which seeks to ensure that all eligible staff who have any significant

responsibility to undertake research are returned to the REF. Where deemed necessary, the

funding bodies may seek to verify through audit that HEIs have adhered to the processes

set out in their code of practice.

140.

The indicators set out in this guidance are not intended to be exhaustive, but should

provide an illustration to HEIs to help inform the development of their own processes.

Additionally, the indicators are not intended to define a minimum threshold, and staff may

be identified as having significant responsibility for research without fulfilling all of the

below. However, the funding bodies would not consider an HEI’s process appropriate if that

process identified staff as without significant responsibility for research where the staff fulfil

the majority of the indicators outlined.

141.

Staff with significant responsibility for research are those for whom:

‘Explicit time and resources are made available’. Indicators of this could include:

•

a specific proportion of time allocated for research, as determined in the context

of the institution’s practices and applied in a consistent way

•

research allocation in a workload model or equivalent.

‘To engage actively in independent research’. Indicators of this could include

(HEIs are also advised to refer to the indicators of independence, paragraph 132,

as additional guidance on this aspect):

•

eligibility to apply for research funding as the lead or co-applicant

•

access to research leave or sabbaticals

•

membership of research centres or institutes within the HEI.

‘And that is an expectation of their job role’. Indicators of this could include:

•

current research responsibilities as indicated in, for example, career pathways

or stated objectives

•

expectations of research by role as indicated in, for example, job descriptions

and appraisals.

REF 2021 35

142.

The submission approach may vary by UOA where employment practices vary at this

level. It is expected that variations will be based on disciplinary norms (for example, close

working with industry), rather than purely local differences in practice. The institution’s

criteria for identifying staff should be developed collaboratively with the academic staff

body and evidence of institution-wide consultation on the criteria should be available in

the institution’s code of practice. The criteria must be:

•

objective

•

non-discriminatory

•

transparent.

They must allow for appropriate staff engagement, contain clearly defined responsibilities,

and take due regard of the Public Sector Equality Duty (section 75 in Northern Ireland) (see

REF 2019/03). Evidence of agreement by staff representative groups should be provided.

The processes should be context dependent, drawing on standard ways of working at

the institution, and it should be possible to test these criteria fairly and evenly against the

responsibilities of all academic staff.

143.

Institutions must be able to verify through audit that eligible staff who are not

submitted do not have significant responsibility for research, with the evidence for this

related to the institution’s documented process. Where an audit process identifies staff with

significant responsibility for research that have not been submitted, their FTE will be added

to the unit’s submission and the total output requirement will be recalculated accordingly.

An unclassified score will be added to the outputs sub-profile for each missing output in the

recalculated total.

144.

Where a staff member is removed through audit, all outputs attributed to that

individual will be removed from the unit’s submission and the total output requirement

will be recalculated (see paragraph 137).

36 REF 2021

Figure 1

REF 2021 37

Research staff data requirements (form REF1a)

145.

The following data are required on all Category A submitted staff:

HESA staff identifier. This is for verification and equal opportunities monitoring

purposes.

Staff reference code: a code determined by the HEI (collected only where there is

no HESA staff identifier).

Initials.

Surname.

Date of birth. This is for verification purposes to enable the REF team to uniquely

identify staff.

Open research and contributor ID (ORCID), where held.

Contracted FTE on the census date. The minimum FTE that may be reported is 0.2.

For staff between 0.2 to 0.29 FTE, details of the research connection with the

submitted unit (see paragraphs 123 to 127).

If the individual is on a fixed term contract, secondment, or period of unpaid leave,

the start and end dates (day, month and year) of the contract, secondment or

period of unpaid leave. Staff on rolling contracts or a series of renewable fixed-term

contracts will be regarded as fixed-term for this purpose, although institutions may

wish to draw attention to their use of rolling contracts in the textual part of their

submissions, especially where a fixed-term contract has an expiry date soon after

the census date.

Any research groups that the individual belongs to, where relevant and up to a

maximum of four. This is not a mandatory field. Some sub-panels may ask HEIs

to describe research groups in REF5b (the unit-level environment template), but

neither the presence nor absence of research groups is assumed.

Early career researchers

146.

A request to the funding bodies for a reduction in the submitting unit’s output

requirement may be made in connection with early career researchers (ECRs), as

described in the proposals set out in paragraphs 160 and Annex L. Requests may also be

made for ECRs to be returned without the minimum of one requirement in exceptional

circumstances, as described in paragraphs 178 to 183.

147.

Regardless of whether or not a request for an output reduction is made in connection

with ECRs, all staff included in a submission who meet the definition of an ECR will be

identified as ECRs in the submission through the HESA staff record. This will not affect the

REF 2021 scores. This is to enable the funding bodies to examine any effects for ECRs in the

sector analysis as described in paragraph 37. To enable this analysis, the HESA staff return

38 REF 2021

for 2019–20 will include a field for HEIs to identify all eligible academic staff on ‘research’ or

‘teaching and research’ contracts who meet the REF definition of an ECR.

148.

ECRs are defined as members of staff who meet the definition of Category A eligible

on the census date, and who started their careers as independent researchers on or after

1 August 2016. For the purposes of the REF, an individual is deemed to have started their

career as an independent researcher from the point at which:

they held a contract of employment of 0.2 FTE or greater, which included a primary

employment function of undertaking ‘research’ or ‘teaching and research’, with any

HEI or other organisation, whether in the UK or overseas, and

they first met the definition of an independent researcher (paragraphs to 131

to 133).

149.

The following do not meet the definition of an ECR (this list is not exhaustive):

Staff who first acted as an independent researcher while at a previous employer –

whether another HEI, business or other organisation in the UK or elsewhere –

before 1 August 2016, with a contract of 0.2 FTE or greater.

Staff who first acted as an independent researcher before 1 August 2016 and have

since had a career outside of research or an extended break from their research

career, before returning to research work. Career breaks outside the HE sector are

included in the types of circumstances where requests for output reductions may

be made (see paragraph 160).

Research assistants who would not normally meet the definition of an independent

researcher, as set out in paragraph 129.

Data requirements for former staff (form REF1b)

150.

Outputs in the submitted output pool may be attributed to former staff, previously

employed as Category A eligible in the assessment period. Full guidance on the eligibility of

these outputs is set out in paragraphs 211 to 216. Where such outputs are submitted, the

following data on the former staff to whom they attributed in the submission are required:

HESA staff identifier (where held). This is for verification purposes.

Staff reference code: a code determined by the HEI (collected only where there is

no HESA staff identifier).

Initials.

Surname.

Date of birth. This is for verification purposes to enable the REF team to uniquely

identify staff.

ORCID, where held.

REF 2021 39

For each eligible employment period where an output is being claimed:

FTE of REF-eligible contract(s).

For staff between 0.2 to 0.29 FTE, details of the research connection with the

submitted unit (see paragraphs 123 to 127).

Start and end dates (day, month, year) of the REF-eligible contract(s).

If the individual was on a period of secondment, or unpaid leave when any outputs

attributed to them were first made publicly available, the start and end dates

(day, month and year) of the secondment or period of unpaid leave.

Any research groups that the individual belonged to, where relevant and up to a

maximum of four. This is not a mandatory field. Some sub-panels may ask HEIs

to describe research groups in REF5b (the environment template), but neither the

presence nor absence of research groups is assumed.

Staff circumstances

151.

The UK funding bodies are committed to supporting and promoting equality and

diversity in research careers. As part of this commitment, the measures set out in this

section have been put in place to recognise the effect that individuals’ circumstances may

have on research productivity.

152.

Part 3, Section 2 of this document sets out the requirements for the submitted output

pool. The total number of outputs returned from each submitting unit must be equal to

2.5 times the combined FTE of Category A submitted staff included in the submission.

A minimum of one output will be required for each Category A submitted staff member.

There will be no minimum requirement for submitting the outputs of former staff. No more

than five outputs may be attributed to any individual staff member (including former staff).

153.

The decoupling of staff and outputs in REF 2021, as summarised in paragraph 152, is

intended to provide increased flexibility to institutions in building the portfolio of outputs

for submission. There are many reasons why an excellent researcher may have fewer or

more outputs attributable to them in an assessment period. It is therefore not expected that

all staff members would be returned with the same number of outputs attributed to them

in the submission.

154.

It is anticipated that the flexibility described above will be reflected in institutions’

expectations of individual researchers. Institutions must set out in their code of practice

how their output selection process takes into account the circumstances of individual

researchers and how appropriate support is provided to affected individuals.

155.

The funding bodies have made every effort to try to eliminate any incentives towards

discriminatory practices by HEIs in the process; to the extent that there are any such

inadvertent incentives, it is the HEIs’ responsibility as employers and public bodies to

ensure that they avoid engaging in discriminatory practices.

40 REF 2021

Measures to support staff with individual circumstances

156.

As a key measure to support equality and diversity in research careers, the funding

bodies have put in place processes to recognise the effect that an individual’s circumstances

may have on their productivity. The approach to staff circumstances set out here has been

developed by the funding bodies to meet a number of key principles:

Ensure recognition of the effect circumstances can have upon an individual

researcher’s productivity.

Create the right incentives for HEIs to support staff with circumstances (and avoid

introducing negative incentives, for example around recruitment).

Recognise the potential disparity in the available output pool for units in particular

contexts, for example where there are high proportions of staff with circumstances,

or for very small units.

Maintain the integrity of exercise – both in supporting equality and diversity and

ensuring the credibility of assessment process.

157.

All HEIs participating in REF 2021 will be required to establish safe and robust

processes to enable individuals to declare voluntarily their individual circumstances

and have the impact of those circumstances reflected in the HEI’s expectations of their

contribution to the output pool. These processes must be documented in the institution’s

code of practice.

158.

Where required, submitting units may optionally request a reduction, without penalty,

in the total number of outputs required for a submission. It is expected that requests will

only be made where the cumulative effect of circumstances has disproportionately affected

the unit’s potential output pool. The reductions applied should be set out in accordance with

the guidance set out below.

159.

In addition, in all UOAs, an individual may be returned without the required minimum

of one output without penalty in the assessment, where the nature of the individual’s

circumstances has had an exceptional effect on their ability to work productively

throughout the period, so that the staff member has not been able to produce the required

minimum of one output. Further details are set out in paragraphs 178 to 183. This measure

is intended to minimise any potential negative impact on the careers of particular groups

of researchers who have not been able to produce an output in the period due to their

individual circumstances.

Summary of applicable circumstances

160.

The funding bodies, advised by EDAP, have identified the following equality-related

circumstances that, in isolation or together, may significantly constrain the ability of

submitted staff to produce outputs or to work productively throughout the assessment

period. Details of the permitted reductions are set out in Annex L:

Qualifying as an ECR (on the basis set out in paragraphs 148 and 149 and

Annex L).

REF 2021 41

Absence from work due to secondments or career breaks outside the HE sector.

Qualifying periods of family-related leave.

Other circumstances that apply in UOAs 1–6, as defined in paragraphs 162 to 163.

Circumstances with an equivalent effect to absence, that require a judgement

about the appropriate reduction in outputs, which are:

i. Disability: this is defined in the ‘Guidance on codes of practice’,

Table 1 under ‘Disability’.

ii.

Ill health, injury, or mental health conditions.

iii.

Constraints relating to pregnancy, maternity, paternity, adoption or childcare

that fall outside of – or justify the reduction of further outputs in addition to –

the allowances set out in Annex L.

iv. Other caring responsibilities (such as caring for an elderly or disabled

family member).

v. Gender reassignment.

vi. Other circumstances relating to the protected characteristics listed in the

‘Guidance on codes of practice’, Table 1, or relating to activities protected by

employment legislation.

161.

As part-time working is taken account of within the calculation for the overall number

of outputs required for the unit (which is determined by multiplying the unit’s FTE by

2.5) reduction requests on the basis of part-time working hours should only be made

exceptionally. For example, where the FTE of a staff member late in the assessment period

does not reflect their average FTE over the period as a whole.

162.

In UOAs 1–6, the number of outputs may be reduced by up to one, without penalty

in the assessment, for Category A submitted staff who are junior clinical academics. These

are defined as clinically qualified academics who are still completing their clinical training in

medicine or dentistry and have not gained a Certificate of Completion of Training (CCT) or its

equivalent prior to 31 July 2020.

163.

This allowance is made on the basis that the clinical staff concerned are normally

significantly constrained in the time they have available to undertake research during the

assessment period. Where the individual meets the criteria in paragraph 162, and has had

significant additional circumstances – for any of the other reasons in paragraph 160 – the

institution can make a case for further reductions as part of the unit reduction request,

using the tariffs set out in Annex L as a guide.

HEIs’ processes

164.

The funding bodies expect all HEIs participating in the exercise to put in place safe and

supportive processes to enable staff to declare voluntarily their individual circumstances,

and to recognise the effect of those circumstances on a staff member’s ability to contribute

to the output pool at the same rate as other staff.

42 REF 2021

165.

To aid institutions in promoting equality, complying with legislation and avoiding

discrimination, institutions must document in a code of practice and apply fair and

transparent processes for the selection of outputs. Further information can be found in

the ‘Guidance on codes of practice’ (REF 2019/03). The funding bodies may seek to verify

through audit that HEIs have adhered to the approaches set out in their code of practice.

Supporting staff declaration

166.

It is the funding bodies’ view that the individual staff member is best placed to consider

whether equality-related circumstances (as set out in paragraph 160) have affected their

productivity over the REF assessment period and that they should not feel under pressure

to declare their circumstances where they do not wish to do so. It is also important

to ensure that processes are applied equally to all applicable circumstances, whether

previously known to the institution or first identified through the staff circumstances

process. Therefore, submitting institutions should not take account in the REF submission

process of any individual circumstances other than those that staff have consented to

declare voluntarily.

167.

Submitting institutions will need to develop robust processes that support staff

to declare individual circumstances in a consistent way, with an appropriate degree of

confidentiality. Particular regard should be paid to the declaration of sensitive issues such

as ongoing illness or mental health conditions.

168.

The process for supporting staff to declare circumstances should be conducted

proactively: instead of relying on individuals coming forward, staff should be invited to

complete a form about their individual circumstances, and provided with clear information

about the applicable circumstances (as set out in paragraphs 160 to 163) and how the

declaration process will operate. HEIs should document in their code of practice how this

process will be undertaken. It should be made clear to staff that they are not required to

complete and return this form where they do not wish to do so. To support institutions with

this process, we will provide a template declaration form that could be used by institutions

for this purpose.

169.

Responding to concerns raised in the 2018 consultation on the ‘Draft guidance on

submissions’, the funding bodies require that institutions’ processes include measures

to guard against undue pressure being placed on staff to declare circumstances, either

centrally or at unit level. Institutions should consider where the process should be managed

(i.e. centrally or at faculty/departmental level), who should be involved in the process, and

what guidance should be provided both to those involved in the process and to those in

management positions within submitting units.

Recognising the effect of staff circumstances

170.

A key aim of the measures is to give recognition to the effect that individual

circumstances may have on a researcher’s ability to contribute to the unit’s overall output

pool (from which submitted outputs will be selected). The funding bodies consider that this

recognition is a core part of a research environment that promotes equality and diversity,

and that responsibility for this rests with institutions.

REF 2021 43

171.

Institutions will have different approaches to preparing REF submissions, and it is not

appropriate for one set approach to be prescribed. Therefore, an HEI’s code of practice will

need to describe how, in the context of its own approach to preparing submissions, the

institution will adjust its expectations about staff contributions to the overall output pool

where circumstances are declared.

172.

The processes for supporting staff with circumstances should be consistent across

the institution. Careful consideration should be given to the nature and timing of support

offered and any adjustments to expectations should be made in consultation with the

individual affected. In developing processes to support staff and adjust expectations,

institutions are encouraged to be guided by the tariffs in place for requesting reductions to

the output requirement (see Annex L). Further guidance is set out in the ‘Guidance on codes

of practice’.

Assessing the effect on the overall output pool

173.

Submitting units may wish to consider the cumulative effect of staff circumstances on

the unit’s overall output pool and whether a request should be made for a reduction to the

total number of outputs required for the submission.

174.

In view of the flexibility offered by decoupling, and the reduction in output requirement

since the previous exercise – from four outputs per person in REF 2014 to an average of 2.5

per FTE in REF 2021 – it is the funding bodies’ view, supported by advice from EDAP, that

institutions will not routinely need to request reductions to the number of outputs required

by a submitting unit. The measures set out above (paragraphs 166 to 172) are considered

an effective way to recognise the effect of circumstances on individual staff productivity and

ensure the aim of promoting equality and diversity is met.

175.

It is clear that there are some instances, however, whereby the available output pool

for a given unit has been disproportionately affected by equality-related circumstances. In

such instances, it would be appropriate for the institution to seek a reduction to the total

number of outputs required for that submitting unit. Such cases may include where there

are very high proportions of staff in the unit whose individual circumstances have affected

their productivity over the REF assessment period, including in very small units, or where

disciplinary publishing norms make it likely that an individual will have generated a smaller

number of outputs across the publication period.

176.

HEIs are required to set out in their code of practice their processes for determining

where a reduction request is necessary. The funding bodies consider that the size of the

available output pool (from which selection will be made) in terms of its proximity to the

total number of outputs required would be one useful indicator for determining whether a

reduction request should be made.

177.

Where making requests, submitting units should apply the tariffs set out in Annex L.

Requests must be accompanied by a supporting statement that includes information on

the context of the unit (for example, size, proportion of those with declared circumstances),

how the circumstances affected the unit’s output pool and why this was determined to be

disproportionate, and how this complies with the process set out in the institution’s code

of practice.

44 REF 2021

Removing the ‘minimum of one’ requirement

178.

All Category A submitted staff must be returned with a minimum of one output

attributed to them in the submission, including staff with individual circumstances.

However, where an individual’s circumstances have had an exceptional effect on their ability

to work productively throughout the assessment period (1 January 2014 to 31 July 2020),

so that the individual

has not been able to produce an eligible output

, a request may be

made for the minimum of one requirement to be removed (form REF6a). Where the request

is accepted, an individual may be returned with no outputs attributed to them in

the submission, and the total outputs required by the unit will be reduced by one.

179.

Requests may be made for an individual researcher who has not been able to

produce an eligible output where any of the following circumstances apply within the period

1 January 2014 to 31 July 2020:

an overall period of 46 months or more absence from research during the

assessment period, due to one of more of the circumstances set out in paragraphs

160 to 163 (such as an ECR who has only been employed as an eligible staff

member for part of the assessment period)

circumstances equivalent to 46 months or more absence from research, where

circumstances set out in paragraph 160 apply (such as mental health issues, caring

responsibility, long-term health conditions) or

two or more qualifying periods of family-related leave, as defined in Annex L.

180.

Where the circumstances cases do not apply, but the individual’s circumstances are

deemed to have resulted in a similar impact (including where there are a combination of

circumstances that would not individually meet the thresholds set out), a request may still

be made and the institution should clarify this within the request form. Where an individual

has a combination of circumstances, all the applicable circumstances should be cited in the

request and information provided about the effect of the combined circumstances on the

researcher’s ability to produce an eligible output in the period.

181.

The rationale for including two or more qualifying periods of family-related leave is

based on the funding bodies’ and EDAP’s considered judgement, informed by the REF expert

panels, that the impact of two or more periods of such leave may be sufficiently disruptive

of an individual’s research that they have not been able to produce an eligible output.

182.

The request should include a description of how the circumstances have affected the

staff member’s ability to produce an eligible output in the period. The information provided

in the request must be based on verifiable evidence, which may be audited in 2021,

following the REF submission deadline.

10.

This may include absence from work due to working part-time, where this has had an exceptional effect

on ability to work productively throughout the period 1 January 2014 to 31 July 2020, so that the individual

has not been able to produce an eligible output. For part-time working, the equivalent ‘total months absent’

should be calculated by multiplying the number of months worked part-time by the full- time equivalent

(FTE) not worked during those months. For example, an individual worked part-time for 30 months at 0.6

FTE. The number of equivalent months absent = 30 x 0.4 = 12.

REF 2021 45

183.

Where a request is agreed, one output will be removed from the total output pool

required for the submitting unit. This will be in addition to any reduction (of up to 1.5

outputs) applied for that staff member in REF6b, according to the guidance set out in

paragraphs 186 to 191. If the staff member concerned moves institution before or on the

census date, the removal of the minimum of one requirement may be applied by the newly

employing institution.

Unit reduction requests

184.

Reduction requests will be made at the level of the submitting unit, and should set out

the requested reduction to the total number of outputs required for that unit. This section

provides further details about calculating requests, what the data requirements are, and

how requests should be made.

185.

Where a unit has not submitted a reduction request and is returned with fewer than

2.5 outputs per FTE, and/or has not attributed a minimum of one output to each Category A

submitted staff member, any ‘missing’ outputs will be graded as ‘unclassified’.

Calculating reductions

186.

For REF6b, the total reduction should be calculated as a sum of the reductions arising

from individual staff in the unit with applicable circumstances, which have constrained their

ability to work productively during the assessment period. This should be calculated within

the following guidance:

Reductions arising from the circumstances of individual staff should be determined

according to the tables and guidance in Annex L, up to a reduction of 1.5 outputs

per staff member affected. This should include reductions, up to 1.5 outputs,

arising from the circumstances of staff for whom a request is also being made to

remove the minimum of one requirement (see paragraphs 178 to 183).

Where multiple circumstances have affected individual staff, the guidance on

combining circumstances in Annex L should be followed.

The reductions applying to individual staff in the unit should be summed. Rounding

to the nearest whole number should be applied to give a whole number of outputs

for reduction.

187.

For each member of staff for whom a request is made in REF6a to remove the

minimum of one requirement (as set out in paragraphs 178 to 183), a further reduction of

one output will be added to the unit request. The combined total across REF6a and 6b will

set out the unit’s requested reduction.

188.

HEIs must ensure that the proposed reduction would not result in a smaller total

output requirement than the number of Category A submitted staff in the unit for whom

a minimum of one output is required.

189.

When making a request institutions may take account of where an individual’s

circumstances are ongoing at the point of making the request, in applying defined

reductions, making a judgement on circumstances equivalent to absence, or removing the

minimum requirement,.

46 REF 2021

190.

In order to provide clarity and consistency on the number of outputs that may be

reduced without penalty where a reduction request is being made, we have set out defined

reductions for staff circumstances (as listed in Annex L). Additionally, the application of

reductions (including those that will require a judgement) will be considered on a consistent

basis in advance of the census date.

191.

The guidance set out in Annex L will apply across all UOAs, and where a reduction is

applied, the remaining number of submitted outputs will be assessed without any penalty.

Data requirements for unit reduction requests and requests to remove the minimum

of one requirement (REF6a/b)

192.

For each member of staff for whom a request to remove the minimum of one

requirement is being made, the following information must be provided in REF6a:

Information to enable the REF team to identify the staff member within

the submission.

Details about which circumstances listed in paragraph 160 (or whether

different circumstances) apply.

A brief statement (max. 200 words) describing how the circumstances have

affected the staff member’s ability to produce an eligible output in the period.

193.

For each unit reduction request, the following information must be provided in REF6b:

Details about the number of staff in the unit with each of the defined circumstances

and information that will enable the REF team to identify these staff members

within submissions (including the HESA ID).

For each member of staff with circumstances requiring a judgement, information

to enable the REF team to identify the staff member within the submission, a brief

outline (max. 200 words) of the nature of the circumstances and how the HEI

determined an appropriate reduction, and the reduction proposed.

A supporting statement (max. 300 words) outlining the rationale for requesting a

unit reduction in accordance with the HEI’s code of practice.

194.

The information returned in REF6a/b for any type of circumstances must be based

on verifiable evidence (although, for the avoidance of doubt, the REF typically will accept

individuals’ self-descriptions of their circumstances and institutions are expected to

approach any verification of circumstances with tact and care and only in accordance with

the law).. EDAP may request further details during the process of reviewing the request, and

all submitted information will be subject to audit during the assessment year (2021).

195.

Information submitted in requests will be kept confidential to the REF team, EDAP and

the main panel chairs, who are all subject to confidentiality undertakings in respect of all

information contained in submissions. REF sub-panels will know where a reduction in the

overall number of outputs in the submitting unit has been agreed without penalty on the

basis of individual circumstances, but will not have access to further information about the

REF 2021 47

circumstances. These arrangements will enable individuals to declare the information in

a confidential manner, and enable consistent treatment of individual circumstances across

the exercise.

196.

Information submitted in requests will be used only for the respective purposes

of considering: requests for a reduction in the number of outputs required from the

submitting unit overall, and for removing the minimum of one requirement for an individual

staff member. This information will not be published and will be destroyed on completion of

the REF in December 2021.

197.

It is the responsibility of the HEI to ensure that the personal data in requests is

submitted in compliance with current data protection legislation – General Data Protection

Regulation (EU) 2016/679 and the Data Protection Act 2018 – and all other legal obligations.

Request process

198.

In autumn 2019, we will invite institutions to submit unit reduction requests and

requests for removing the minimum of one requirement; the deadline for submitting

requests will be March 2020. Requests will be submitted via the secure submission system.

The outcome of requests will be provided before the census date.

199.

Where there are changes to the Category A submitted staff employed in the unit after

the request has been submitted, institutions will be able to amend REF6a/b and make

requests for further reductions at the point of submission, as follows:

Staff included in the unit request who are no longer employed in the unit on the

census date should be removed from the request, and the approved reduction

recalculated accordingly.

Any applicable circumstances of staff who join the unit on or before the census

date, but following the original request submission, may be added to the request

form. In these cases, HEIs should include the information outlined in paragraph 192

a.-c. and/or paragraph 193 a.-b., as relevant. Decisions in these instances will be

taken during the assessment year – with respect to the new details only.

200.

All unit reduction requests will be considered by EDAP on a consistent basis across all

UOAs. EDAP may seek advice on submitted requests from the main panel chairs. EDAP will

make recommendations about the appropriate number of outputs that may be reduced

without penalty in the case of circumstances requiring a judgement. EDAP will also make

recommendations on accepting requests for the removal of the minimum of one. Where

required, EDAP may request further information to confirm the correct application of tariffs.

EDAP will provide a written explanation where a request is not accepted in full, or in part.

The membership and terms of reference of EDAP are available at www.ref.ac.uk under

‘Equality and Diversity’.

201.

An appeals process will be in place to consider any concerns an institution has with

respect to the processes followed in determining reductions. In this event, the institution

should contact the REF team in the first instance. Full details of the appeals process will be

provided to institutions when we invite institutions to submit circumstances requests.

48 REF 2021

Part 3 Section 2: Research outputs (REF2)

202.

Each HEI must decide which outputs to select for submission, in accordance with the

following guidance and its internal code of practice (see REF 2019/03).

203.

The decoupling of staff and outputs in REF 2021 is intended to provide increased

flexibility to institutions in building the portfolio of outputs for submission. There are many

reasons why an excellent researcher may have fewer or more outputs attributable to

them in an assessment period. It is therefore not expected that all staff members would

be returned with the same number of outputs attributed to them in the submission. As

set out in REF 2019/03, to aid institutions in promoting equality, complying with legislation

and avoiding discrimination, institutions must document and apply fair and transparent

processes for the selection of outputs. These must demonstrate how they have taken into

account equality and diversity considerations, and any equality-related circumstances

affecting staff ability to research productively during the period.

204.

As part of the standard analyses provided to panels (see Annex

J),

we will include

data on the distribution of outputs attributed to staff in submissions. There will be an

opportunity in the unit-level environment template (REF5b) for institutions to contextualise

the distribution of outputs, drawing on the processes described in the code of practice.

Eligibility definitions for research outputs

205.

Submissions must include a set number of items of research output, equal to 2.5 times

the combined FTE of Category A submitted staff included in the submission. Rounding to the

nearest whole number will be applied to give a whole number of outputs for submission

This number will be adjusted, as appropriate, to take account of successful requests for staff

circumstances (see paragraphs 151 to 201). Each output must be:

The product of research, briefly defined as a process of investigation leading to new

insights, effectively shared. (The full definition of research for the purposes of the

REF is in Annex C.)

First brought into the public domain during the publication period 1 January 2014

to 31 December 2020 or, if a confidential report, lodged with the body to whom it

is confidential during this same period (see paragraphs 261 to 263).

Attributable to a current or former member of staff, who made a substantial

research contribution to the output, which must be either:

i. produced or authored solely, or co-produced or co-authored, by a Category

A submitted staff member, regardless of where the member of staff was

employed at the time they produced that output, or

ii. produced or authored solely, or co-produced or co-authored, by a former staff

member who was employed by the submitting HEI according the Category A

eligible

definition when the output was first made publicly available.

11.

Values ending in .5 should be rounded up.

12.

With the exception of being employed on the census date.

REF 2021 49

Available in an open access form, where the output is within scope of the open

access policy (see paragraph 223).

206.

A summary of output eligibility is set out in Figure 2.

207.

The submitted pool of outputs should include:

A minimum of one output for each Category A submitted staff member, which has

been produced or authored solely, or co-produced or co-authored, by that staff

member (unless individual circumstances apply).

Further outputs up to the total required for the submitting unit, taking into account

any applicable reductions for staff circumstances. A maximum of five outputs may

be attributed to an individual staff member (both Category A submitted staff, as

well as any former staff whose outputs are eligible for submission). The attribution

of the maximum number of outputs to a staff member will not preclude the

submission of further outputs on which that staff member is a co-author, where

these are attributed to other eligible staff in the unit.

208.

A diagram illustrating the submitted output pool requirements is set out in Figure 3.

209.

Outputs may only be attributed to individuals who made a substantial research

contribution to the output. This information will be made available to panels to enable them

to establish whether a substantial research contribution has been made. The ‘Panel criteria’

sets out whether the panels require any additional information for co-authored outputs.

210.

Outputs determined to be ineligible through audit will be removed from the

submission and an unclassified score added to the profile to account for the ‘missing’

output. Where this involves removing the only output associated with a Category A

submitted staff member, the REF team may audit the eligibility of the staff member, and

review the submitted FTE accordingly.

Eligibility of outputs produced or authored by former staff

211.

The introduction of a transitionary approach to non-portability of outputs will allow

a submitting unit to include the outputs of staff formerly employed as Category A eligible

(former staff). Outputs attributable to these staff are eligible for inclusion where the output

was first made publicly available while the staff member was employed by the institution as

a Category A eligible member of staff. This includes:

For staff who remain employed at the institution, but are no longer employed as

Category A eligible staff on the census date (for example, senior administrative

staff), any outputs that were first made publicly available at the point the staff

member was employed as Category A eligible.

Any outputs first made publicly available while a former staff member was on an

unpaid leave of absence or secondment (whether to another UK HEI, or beyond HE/

overseas), where the leave or secondment period was no greater than two years.

50 REF 2021

212.

The outputs of staff who continue to be employed by the institution as Category

A eligible staff (i.e. meet the criteria set out in paragraph 117) but who no longer have

significant responsibility for research on the census date are not eligible.

213.

A former staff member may not have outputs attributed to them in more than one

submission by the institution. Where an individual held a joint appointment across two

or more submitting units within the same institution, the HEI will need to decide on one

submission in which to return any outputs attributed to that individual.

214.

Where an output is first made publicly available in the REF 2021 publication period

in both pre-published (such as online first, or pre-prints) and in final form, and the author

moved institution in the intervening period, the institution employing the staff member

when the pre-published version of the output was made first publicly available should

submit the final version, where possible. Where it is not possible to identify the final version

(for example, for some practice research outputs), the institution should submit the version

that was made publicly available when the member of staff was employed at the HEI.

215.

Outputs of former staff are only eligible where they were fi made publicly available in

the period when the staff member was employed by the submitting institution as a Category

A member of staff Any outputs fi made publicly available in the period preceding or

following this will not be eligible for submission, except in the case set out in paragraph 214.

The funding bodies may seek to verify through audit that the outputs of former staff meet

the eligibility criteria, including the timeframe and staff eligibility requirements.

216.

Outputs that are first made publicly available in the final months of the publication

period (August to December 2020), are only eligible for submission by an HEI that employs

the staff member as Category A eligible on the census date.

General eligibility of outputs

217.

In addition to printed academic work, research outputs may include, but are not limited

to: new materials, devices, images, artefacts, products and buildings; confi or technical

reports; intellectual property, whether in patents or other forms; performances, exhibits or

events; and work published in non-print media. An underpinning principle of the REF is that

all forms of research output will be assessed on a fair and equal basis. Sub-panels will not

regard any particular form of output as of greater or lesser quality than another per se.

A glossary of output categories and collection formats is attached in Annex K.

218.

Institutions that wish to submit outputs produced in the medium of Welsh are welcome

to do so. Such outputs will be assessed equitably, as described in paragraphs 285 and 286.

219.

Reviews, textbooks or edited works (including editions of texts and translations) may

be included if they embody research as defined in Annex C. Editorships of journals and

other activities associated with the dissemination of research findings should not be listed

as an output on REF2.

220.

Where two or more research outputs in a submission include significant material in

common (for example, a journal article that also appears as a chapter in a book) the sub-

panels will assess each output taking account of the common material only once. Where

REF 2021 51

a sub-panel judges that they do not contain sufficiently distinct material and should be

treated as a single output, an unclassified score would be given to the ‘missing’ output.

221.

Theses, dissertations or other items submitted for a research degree including doctoral

theses may not be listed. Other assessable published items based on research carried out

for a research degree may be listed.

222.

HEIs may not submit any output produced by a research assistant or research student

supervised by a Category A eligible staff member employed in the unit, unless the staff

member co-authored or co-produced the output.

52 REF 2021

Figure 2

REF 2021 53

Figure 3

54 REF 2021

Open access

The scope of this policy

223.

The requirement to comply with the open access policy applies to the following outputs

that are listed in REF2:

the output type is a journal article with an ISSN or the output is a conference

contribution in conference proceedings with an ISSN

and

the date of acceptance of the output for publication (see paragraph 227) is after

1 April 2016.

224.

Any submitted output that fits

both aspects

of this definition is an ‘in-scope’ output.

Other than the exception in paragraph 231, in-scope outputs must fulfil the open access

criteria set out below to be eligible for submission.

225.

Any output meeting the wider eligibility criteria, but that does not meet both aspects of

this definition, may be listed in REF2 without meeting the open access policy requirement.

For clarity, the open access requirement does not apply to output types such as:

•

monographs and other long-form publications

•

non-text outputs

•

working papers or outputs submitted to pre-print systems that are not the version

‘as accepted for publication’

•

the data which underpins some research

•

confidential reports that are not published because of commercial or

other sensitivity.

226.

The policy requirement does not apply to outputs underpinning research impact.

Key terms

227.

‘Date of acceptance’ means the date given in the acceptance letter or email from the

publisher to the author as the ‘firm’ accepted date.

228.

Outputs that are published by a journal or conference proceedings which do not

require peer review are within the scope of this policy. In this instance, the author’s final

accepted version must be deposited. The date of acceptance in this instance should be

taken as the date that the publisher confirms that the article has been received from the

author and will subsequently be published.

229.

‘Date of publication’ means the date that the final ‘version of record’ is first made

publicly available (such as on the publisher’s website). This will usually mean that the ‘early

online’ date, rather than the print publication date, should be taken as the date

of publication

230.

Author’s accepted manuscript refers to the final peer-reviewed text which may

otherwise be known as the ‘author manuscript’ or ‘final author version’ or ‘post-print’.

13.

If this date is imprecise, for example MM/YYYY, the final day of the given month can be used as date

of publication.

REF 2021 55

Tolerance of non-compliance

231.

For each submission to a unit of assessment, units may submit a maximum of five per

cent non-compliant in-scope outputs, or one non-compliant in-scope output, whichever is

higher, per submission. HEIs will be able to review the percentage of listed, in-scope outputs

that they have identified as not compliant or as having an applicable exception, for each

submission prior to the submission deadline. If an institution wishes to proceed with a

submission exceeding the tolerated proportion/number of non-compliant in-scope outputs,

the institution will be invited to identify which outputs should be removed as ineligible.

An unclassified score will be added for the removed (‘missing’) outputs. Where an audit

process demonstrates that outputs identified as compliant do not meet the open access

requirements and exceed the tolerated proportion/number, these outputs will be removed,

and an unclassified score added for the ‘missing’ outputs. The maximum of five per cent

of outputs will be rounded to the nearest whole output number. There is no minimum

threshold of in-scope outputs in applying the tolerance band.

232.

Other than as set out in paragraph 231, all in-scope outputs must fulfil the open access

criteria set out in paragraphs 234 to 251, or have an applied exception.

233.

Policy exceptions are detailed in paragraphs 252 to 255, and include: deposit

exceptions, access exceptions, technical exceptions, further exceptions. Where an output is

submitted to REF2 with an exception, HEIs should identify that an exception applies. There

will not be a limit on the volume of exceptions to the policy for in-scope outputs submitted.

Use of exceptions will not affect REF outcomes.

Criteria for open access

234.

The criteria consist of three elements:

Deposit requirements

Discovery requirements

Access requirements.

Deposit requirements

235.

The output must have been deposited in an institutional repository

, a repository

service shared between multiple institutions, or a subject repository

14.

Institutions which are submitting under 250 journal articles or conference contributions, or where these

output types comprise less than 50 per cent of their total submitted outputs for REF 2021, can use

institutional webpages to meet the policy requirements.

15.

Individuals depositing their outputs in a subject repository are advised to ensure that their chosen repository

meets the requirements set out in paragraphs 234 to 251

in this policy. REF 2021 guidance will not certify

the repositories which fulfi policy requirements. Institutions should be assured that if there is repository

failure/circumstances outside of their control which impact on output compliance, REF 2021 will not restrict

submission of outputs. In this case an exception to the policy is applicable (paragraph

254.b

). Institutions’

research information management systems that can support the open access requirements through

repository-like functionality can be thought of as institutional repositories for the purposes of this policy.

56 REF 2021

236.

The output must be deposited within the repository within a specified timeframe,

determined by the date of acceptance:

Outputs accepted for publication from the 1 April 2016 to 31 March 2018.

The output must have been deposited as soon after the point of acceptance as

possible, and no later than three months after the date of publication.

Outputs accepted for publication from the 1 April 2018 to 31 December 2020.

The output must have been deposited as soon after the point of acceptance as

possible, and no later than three months after this date.

237.

The output must have been deposited as the author’s accepted manuscript. Where

the published ‘version of record’ is available for deposit within the required timeframe,

and where the journal or conference publisher permits it, the ‘version of record’ may

be deposited instead of the accepted manuscript. Outputs that have been provisionally

accepted for publication, under the condition that the author makes revisions to the

manuscript that result from peer review, are not considered as the final text.

238.

The funding bodies recognise that many researchers derive value from sharing early

versions of papers using a pre-print service. Institutions may submit pre-prints as eligible

outputs to REF 2021 (see Annex K). Only outputs which have been ‘accepted for publication’

(such as a journal article or conference contribution with an ISSN) are within the scope

of the REF 2021 open access policy. To take into account that the policy intent for ‘open

access’ is met where a pre-print version is the same as the author-accepted manuscript, we

have introduced additional flexibility into the open access requirement: if the ‘accepted for

publication’ text, or near final version, is available on the pre-print service, and the output

upload date of the pre-print is prior to the date of output publication, this will be considered

as compliant with the open access criteria (deposit, discovery, and access).

239.

Some UK funders have a preference for gold open access. ‘Gold’ open access usually

means the immediate, permanent, and free to access availability of the published version

of record on the publisher’s website and with a licence that permits copying and reuse.

Outputs that are made open access through the ‘gold’ route, at the point of first publication,

in accordance with other funder’s requirements and definitions, meet the requirement

of the REF 2021 open access policy. HEIs will need to confirm that outputs were available

immediately after publication via the gold route.

240.

Where a deposited output is later replaced or augmented following its initial deposit,

the updated manuscript may be deposited in place of the originally deposited output.

Discovery requirements

241.

The output must be presented in a way that allows it to be discovered by readers and

by automated tools such as search engines. The discovery requirements should typically be

fulfilled through the storage and open presentation of a bibliographic or metadata record in

the repository. Once discoverable, the output should remain so.

242.

Where a deposited output is later replaced or augmented following its initial deposit,

this must also meet the discovery requirements.

REF 2021 57

Access requirements

243.

The output must be presented in a form that allows anyone with internet access

to search electronically within the text, read it and download it without charge, while

respecting any constraints on timing (as detailed in paragraphs 246 to 251). It is advised that

outputs licensed under a Creative Commons Attribution Non-Commercial Non-Derivative

(CC BY-NC-ND) licence would meet the minimum requirement. Once accessible, the output

should remain so.

244.

Outputs whose text is encoded only as a scanned image do not meet the requirement

that the text be searchable electronically.

245.

Where a deposited output is later replaced or augmented following its initial deposit,

this must also meet the access requirements. Embargo periods may not re-start with

subsequent deposits: they are linked to the date of publication.

Timing of compliance with access requirements and embargo periods

246.

The policy allows authors to respect embargo periods set by publishers of: up to 12

months for Main Panels A and B; or 24 months for Main Panels C and D. Outputs that are

under embargo at the submission deadline are compliant with the policy requirements

(provided that the embargo lengths are within the policy requirements).

247.

The required timing of compliance with the access requirements depends on whether

an embargo period is specified.

248.

The policy allows authors to respect embargo periods set by publishers, within

specified maximum periods. Where a publisher specifies an embargo period, an output

will be compliant with the policy requirements where it is deposited as a ‘closed’ deposit.

Closed deposits must meet the deposit and discovery requirements (unless an exception

applies). The full text should be available to read and download after the embargo period

has elapsed.

249.

The routes to determine the timing of compliance with the access requirements are set

out below:

Route 1: For outputs deposited with no or ‘zero’ embargo. Where the output

has no or a ‘zero’ embargo period it must meet the access requirements as soon as

possible and no later than one month after deposit.

Route 2: For outputs deposited under embargo. Where the output is deposited

under embargo, it must meet the access requirements as soon as possible and no

later than one month after the end of the embargo period. The embargo period

typically begins at the point of first publication (including ‘early online’ publication).

250.

Embargo periods should not exceed the following maxima:

12 months for Main Panel A and Main Panel B.

24 months for Main Panel C and Main Panel D.

58 REF 2021

251.

Interdisciplinary research outputs returned in a UOA in Main Panel A or B that span

boundaries with a UOA in Main Panel C or D may respect the longer of the two embargo

periods. The interdisciplinary identifier should be applied for these outputs (see paragraphs

273 to 274).

Exceptions to the open access requirements

Deposit exceptions

252.

The following exceptions deal with cases where the output is unable to meet the

deposit requirements. In the following cases, the output will not be required to meet any

of the open access criteria (deposit, discovery or access requirements).

At the point of acceptance, it was not possible to secure the use of a repository.

There was a delay in securing the final peer-reviewed text (for instance, where

a paper has multiple authors).

The staff member to whom the output is attributed was not employed on a

Category A eligible contract by a UK HEI (defined in paragraphs 52 to 63) at the

time of submission for publication.

It would be unlawful to deposit, or request the deposit of, the output.

Depositing the output would present a security risk.

Access exceptions

253.

The following exceptions deal with cases where deposit of the output is possible, but

there are issues to do with meeting the access requirements. In the following cases, the

output will still be required to meet the deposit and discovery requirements, but not the

access requirements. A closed-access deposit, where allowed, will be required.

The output depends on the reproduction of third-party content for which open

access rights could not be granted (either within the specified timescales, or at all).

The publication concerned requires an embargo period that exceeds the stated

maxima, and was the most appropriate publication for the output.

The publication concerned actively disallows open-access deposit in a repository,

and was the most appropriate publication for the output.

Technical exceptions

254.

The following exceptions deal with cases where an output is unable to meet the criteria

due to a technical issue. In the following cases, the output will not be required to meet the

open access criteria (deposit, discovery or access requirements).

At the point of acceptance, the staff member to whom the output is attributed

was employed at a different UK HEI, and it has not been possible to determine

compliance with the criteria.

REF 2021 59

The repository experienced a short-term or transient technical failure that

prevented compliance with the criteria (this should not apply to systemic issues).

An external service provider failure prevented compliance (for instance, a subject

repository did not enable open access at the end of the embargo period, or

a subject repository ceased to operate).

Further exceptions

255.

Two further exceptions to the policy are outlined below:

‘Other exception’ should be used where an output is unable to meet the criteria

due to circumstances beyond the control of the HEI, including extenuating personal

circumstances of the author (such as periods of extended leave), industrial

action, closure days, and software problems beyond those listed in the technical

exceptions. If ‘other’ exception is selected, the output will not need to meet the

open access criteria (deposit, discovery or access requirements).

The output was not deposited within three months of acceptance date, but was

deposited within three months of the earliest date of publication. In this instance,

the output will need to meet all other policy requirements. This exception does

not need to be applied retrospectively to outputs compliant with the policy from

1 April 2016 to 1 April 2018 which fulfilled the policy requirements within three

months of publication.

Timing of publication

256.

The relevant date for determining whether or not an output was produced within

the publication period, and hence is eligible for submission, will be the date at which the

submitted output first became publicly available (or, for confidential reports, was lodged

with the relevant body). Where this is near to the start or the end of the publication period

(1 January 2014 and 31 December 2020 respectively) and the actual date at which it became

publicly available is not clear, we may require HEIs to submit evidence of the date it became

publicly available. In particular:

Where the date of imprint on a publication lies outside the publication period

but the actual date of appearance is within the publication period, evidence of

the actual date on which it became publicly available will be required for data

verification purposes, such as a letter from the publisher.

Outputs expected to be made publicly available between the submission date

and the end of the publication period (that is, between 27 November 2020 and 31

December 2020) should be flagged in submissions (see paragraph 265.a); where

only some of the data requirements for those outputs can be supplied, we will

require full details to be submitted by 29 January 2021. HEIs may have to physically

submit any output so flagged for verification purposes. A reserve output may

be submitted for any output that is pending publication (see paragraph 266). An

output expected to be made publicly available after 31 December 2020 should not

be submitted, even if it has been accepted for publication.

60 REF 2021

For web content and electronic corpora, HEIs will need to maintain proof of the

date at which the item became publicly available and of its content at that date, for

example a date-stamped scanned or physical printout or evidence derived from

website archiving services.

For non-text outputs, such as performances, we will require evidence of when the

output was disseminated in the public domain.

For products, we will require evidence of the date when the product was produced

in the form in which it is submitted.

257.

Where an author-accepted manuscript is the version of an output to be first made

publicly available, either that date or the earliest date that the version of record is first made

publicly available will be accepted.

258.

An output first published in its final form during the REF 2021 publication period that

was ‘pre-published’ in the previous publication period – whether in full in a different form

(for example, as a pre-print), or as a preliminary version or working paper – is eligible for

submission to the REF, provided that the ‘pre-published’ output was not submitted to

REF 2014.

259.

Other than the exception described in paragraph 258 above, an output published

during the REF 2021 publication period that includes significant material in common with an

output submitted to REF 2014 is eligible only if it incorporates significant new material. In

these cases:

The panel may take the view that not all of the work reported in the listed output

should be considered as having been issued within the publication period; and if

the previously published output was submitted to REF 2014, the panel will assess

only the distinct content of the output submitted to the REF.

Submissions should explain, where they believe necessary, how far any work

published earlier was revised to incorporate new material.

260.

Other than the exception for outputs pending publication in paragraph 256.b, if an HEI

cannot make available a requested output or provide evidence of its publication within the

publication period that item will be removed from the submission and the ‘missing’ output

awarded a grade of unclassified. There will be no opportunity to submit a substitute item.

Confidential reports

261.

Confidential reports include any item produced for and lodged, in the publication

period, with a company, government body or other research sponsor(s), but which has not

been published because of its commercial or other sensitivity. A confidential report may

only be submitted if the HEI has prior permission from the sponsoring organisation that

the output may be made available for assessment. HEIs will confirm permission has been

secured when they make submissions. If the REF team requests a confidential report for

assessment, the HEI must make it available.

REF 2021 61

262.

Confidential reports will only be shared with the REF team and those involved in

the assessment process. All panel members, advisers, observers and others involved in

the assessment process are bound by a confidentiality agreement. Therefore, it should

be possible for HEIs to submit confidential reports without compromising any duty of

confidentiality upon them. There may be main or sub-panel members who HEIs believe

would have a commercial (or other) conflict of interest in assessing confidential reports.

HEIs will be required to name such individuals when making submissions.

263.

Outputs identified by institutions as confidential will not be listed as part of the

published submissions. Confidential reports submitted to the REF will be destroyed as soon

as no longer required for assessment purposes.

Data requirements for outputs (form REF2)

264.

For each output listed, enough information should be given to enable the REF team and

panels to determine precisely what is being listed, whether it is a product of sole or multiple

authorship or production, in what form it exists and where it may be found. The following

are required for each output:

Output number:

sequentially from one for each output listed in a submission.

This number is for administrative convenience of referencing only.

Date of output:

the calendar year in which the output became publicly available.

For outputs attributed to former staff, additionally the month in which the output

first became publicly available.

Type of output:

Outputs should be categorised into the following broad types

(there will be a number of specific data requirements in common for each output

type; further details of these will be available shortly on the REF website):

i. books (or parts of books)

ii. journal articles and conference contributions

iii. physical artefacts

iv. exhibitions and performances

other documents

vi. digital artefacts (including web content)

vii.

other.

Title of the output:

if the output has no title, a description is required.

265. Each of the following is required where applicable to the output:

Pending publication:

a flag to indicate that the output is due for publication

between the submission deadline and the end of the publication period.

Co-authors:

the number of additional co-authors.

Interdisciplinary research:

a flag to indicate to the sub-panel if the output

embodies interdisciplinary research.

62 REF 2021

Forensic science:

a flag to indicate to the sub-panel if the output embodies

research in forensic science (see paragraphs 275 to 276).

Criminology identifier:

a flag to indicate to the sub-panel if the output embodies

research in criminology (see paragraphs 277 to 278).

The research group

to which the research output is assigned, if applicable. This is

not a mandatory field, and neither the presence nor absence of a research group

is assumed.

Output allocation:

where requested in the ‘Panel criteria’, information to assist in

allocating outputs to appropriate readers.

Request for cross-referral:

a request to the sub-panel to consider cross-referring

the output to another sub-panel for advice (see ‘Panel criteria’, paragraphs 399

to 404).

Request to ‘double-weight’ the output: for outputs of extended scale and scope,

the submitting institution may request that the sub-panel weights the output as

two (see paragraphs 279 to 283).

Additional information:

only where required in the ‘Panel criteria’, a brief

statement of additional information to inform the assessment (see paragraph 284).

Open access:

for in-scope outputs only, whether the output is compliant with the

open access requirements, has an applicable exception, or is not compliant (see

paragraphs 223 to 255).

Supplementary information:

the DOI (or other URL, if no DOI is available) for any

supplementary information published alongside an output.

A brief abstract, for outputs in languages other than English

(see paragraphs

285 to 287).

Confidential output:

whether the output should be omitted from the published

data for specific reasons, such as commercial sensitivity or security.

Outputs pending publication

266.

Institutions may include a ‘reserve’ output for each output expected to be made

publicly available between the submission date and the end of the publication period.

A reserve output will be associated with the specific output which is pending publication.

In submitting a reserve, HEIs must ensure that the requirement to submit a minimum of

one for each Category A submitted staff member is still met, and must ensure that the limit

of five outputs attributed to any one current or former staff member is not exceeded, if the

reserve output is assessed. A reserve output (or an output expected to be made publicly

available between the submission date and the end of the publication period) will not be

counted as the minimum of one for a staff member if it is not assessed.

REF 2021 63

267.

If the submitted output pending publication is not made publicly available on or before

the 31 December 2020, the panel will assess the associated reserve output. A ‘reserve’

output will only be assessed in the event that the output is not made publicly available

within the publication period.

Co-authored/co-produced outputs

268.

For co-authored outputs, the number of other authors will be required. Regardless of

the number of authors listed on an output, a co-authored output listed in a submission will

count as a single output in the assessment. Co-authored/co-produced outputs will not be

counted pro-rata.

269.

Where a co-authored output is eligible for return in different submissions

(whether from the same HEI or different HEIs), the output may be returned in any or all

of these submissions.

270.

In exceptional circumstances, single-weighted co-authored outputs may be returned

more than once within the same submission for submissions made to Main Panel D. The

funding bodies recognise that publication practices vary across disciplines and believe

that a panel-specific approach is appropriate to determining in which instances this would

be admissible. The ‘Panel criteria’ (paragraphs 216 to 236) sets out when outputs may be

submitted more than once in the same submission.

271.

Where there are substantial pieces of co-authored work, reflecting large-scale or

intensive collaborative research within the same submitting unit, and a double-weighting

request has been submitted for the output, institutions may attribute the output to a

maximum of two members of staff returned within the same submission. This output

may be counted as the required minimum of one for each staff member. The inclusion of

any reserve outputs in this instance must be in accordance with the minima and maxima

requirements where the panel does not accept the request for double-weighting.

272.

Further guidance is set out in the ‘Panel criteria’ regarding whether additional

information is required about the contribution of the individual member of staff to a co-

authored output; and, if so, how the panels will take account of this information when

undertaking the assessment.

Interdisciplinary research identifier

273.

For the purposes of the REF, interdisciplinary research is understood to achieve

outcomes (including new approaches) that could not be achieved within the framework of

a single discipline. Interdisciplinary research features significant interaction between two or

more disciplines and/or moves beyond established disciplinary foundations in applying or

integrating research approaches from other disciplines.

274.

Institutions are invited to identify outputs across their submissions that meet the

definition of interdisciplinary research as set out in paragraph 273. Sub-panels will consider

this information in determining the most appropriate means of assessing the output, with

advice from the interdisciplinary advisers. This process is distinct from a request for cross-

referral. The ‘Panel criteria’ (paragraphs 390 to 398) sets out further information about the

processes for assessing outputs identified as interdisciplinary. The IDR identifier will also

64 REF 2021

enable the funding bodies to conduct post-exercise analysis of interdisciplinary research in

REF 2021, enabling them to assess the effectiveness of the additional measures put in place.

Forensic science identifier

275.

Forensic science is science in the service of the justice system. It refers to forensic

applications of the physical, chemical and biological sciences, and also incorporates forensic

applications of computer science and statistics, engineering and the scientific components

of archaeology, anthropology, linguistics, psychology and forensic medicine, and other

sciences applied in the civil and criminal courts, as well as in broader quasi-legal and

regulatory contexts.

276.

Submission of research in forensic science should continue to be returned in the most

appropriate UOA for its assessment, as according to the UOA descriptors (see ‘Panel criteria’,

Part 2). Institutions are invited to identify outputs across their submissions that embody

forensic science research, as defined in paragraph 275. Identifying these outputs will help to

ensure appropriate assessment (for example, via cross-referral, the use of joint assessors,

or existing expertise on the panel in which the outputs are submitted). Additionally, it will

enable panels to review the health of UK research in this field, and will generate an outputs

quality sub-profile for forensic science across the exercise. The identifier will not be used in

the assessment for any purpose beyond these stated uses.

Criminology identifier

277.

Criminology is a multidisciplinary subject concerned with crime, criminals and

criminal justice.

278.

Submission of research in criminology should continue to be returned in the most

appropriate UOA for its assessment, as according to the UOA descriptors (see ‘Panel criteria’,

Part 2). Institutions are invited to identify outputs across their submissions that embody

criminology research, as defined in paragraph 277. Identifying these outputs will help to

ensure appropriate assessment (for example, via cross-referral, the use of joint assessors,

or existing expertise on the panel in which the outputs are submitted). Additionally, it will

enable panels to review the health of UK research in this field, and will generate an outputs

quality sub-profile for criminology across the exercise. The identifier will not be used in the

assessment for any purpose beyond these stated uses.

Double-weighted outputs

279.

Institutions may request that outputs of extended scale and scope be double-weighted

(count as two outputs) in the assessment. The panels provide more information in the ‘Panel

criteria’ about outputs that may merit double-weighting in their discipline areas. Institutions’

requests for double-weighting must be accompanied by a statement of up to 100 words

explaining how the scale and scope of the output satisfies these criteria (see the ‘Panel

criteria’, paragraphs 237 to 247).

280.

No single output may be counted as more than double-weighted (two outputs).

281.

Where requesting an output to be double-weighted, the submitting institution must

reduce the number of outputs listed in the submission by one (unless including a ‘reserve’

output, as described in paragraph 282). The sub-panels will decide whether to double-

weight each output that has been so requested, according to the published criteria. This

REF 2021 65

decision will be separate to the panel’s judgement about the quality of that output. Where

the panel decides to double-weight an output, it will count as two outputs in the submission.

Where the panel does not accept the case for double-weighting, it will count the submitted

output as a single output, and grade the ‘missing’ output as unclassified (unless a ‘reserve’

output is included).

282.

Institutions may include a ‘reserve’ output with each output requested for double-

weighting. Double-weighting requests should be made in accordance with the minima

and maxima requirements for attributing outputs to staff, so that in the event the request

is accepted, or in the event that it is not and the reserve output is instead assessed, the

minimum of one requirement is met for each Category A submitted staff member (unless

individual circumstances apply), and no more than five outputs are attributed to any one

member of current or former staff (noting that, where accepted, a double-weighted output

will count as two attributed outputs to one staff member, unless it is attributed to two staff

members as outlined in paragraph 271).

283.

A reserve output will be associated with the specific output for which double-weighting

has been requested. If the request is declined, the panel will assess the associated reserve

output. A ‘reserve’ output will only be assessed in the event that the panel does not accept

the request for double-weighting.

Additional information

284.

The ‘Panel criteria’ sets out where the panels will require additional information relating

to outputs for the assessment of research in their UOA. The published panel criteria set

out which, if any, of the following types of additional information are required, and provide

further details about the nature of the required information, and the associated word

limits up to a maximum of 300 words. Additional information should

only

be submitted if

specifically requested in the ‘Panel criteria’, otherwise it will be disregarded by the panel.

Any additional information provided should

not

be used to volunteer opinions about the

quality of an output. The types of additional information are:

Factual information about the research questions, methodology or means of

dissemination, where these are not described within the output itself. This applies

to practice-based outputs, for example an exhibition, performance or artefact.

Factual information about the signifi of the output where this is not evident

within the output (for example, if the output has gained external recognition, led to

further developments or has been applied). Citation data may not be included in the

additional information. Where sub-panels make use of citation data as additional

information about the academic signifi of outputs, as described in paragraph

288, the citation data will be provided to panels by the REF team on a consistent basis.

Where the output includes significant material published prior to 1 January 2014,

details of how far the earlier work was revised to incorporate new material (see

paragraph 259.b).

Co-authored or co-produced outputs, details of the contribution to the output of

the staff member to whom it has been attributed in the submission.

66 REF 2021

Outputs in languages other than English

285.

For research outputs in a language other than English (including outputs submitted

in the medium of Welsh), a short abstract in English should be provided to describe the

content and nature of the work (maximum 100 words). A separate field for each output

in REF2 will be available for this. Panels will use this abstract to identify appropriate

assessment, including (where required) external specialist advisers to whom the work may

be referred. The abstracts themselves will not form the basis for assessment. Work may be

referred to external specialist advisers only where panel members and assessors are unable

to assess an output in the language in which it is submitted.

286.

In the case of research outputs in the medium of Welsh, the specialist adviser(s) will

normally be paired with a designated panel member with whom they will discuss the advice

provided. If a sub-panel receives a substantial volume of research outputs in the medium of

Welsh, the specialist adviser(s) will be invited to attend one or more of the panel meetings

during the assessment phase. These provisions are made in recognition of the particular

legal status of the Welsh language in Wales.

287.

The requirement for an abstract is waived for outputs submitted in UOA 26 if the

output is produced in any of the languages within the remit of that UOA; and for any other

UOAs that indicate in their criteria statements that they are able to assess outputs in

that language.

Citation data

288.

Some sub-panels will consider the number of times that an output has been cited

as additional information about the academic significance of submitted outputs. Those

panels that do so will continue to rely on expert review as the primary means of assessing

outputs, in order to reach rounded judgements about the full range of assessment criteria

(‘originality, significance and rigour’). They will also recognise the significance of outputs

beyond academia wherever appropriate, and will assess all outputs on an equal basis,

regardless of whether or not citation data is available for them.

289.

The panels state in the ‘Panel criteria’ if they will make use of citation data, and

if so, provide further details about how they will make use of the data to inform their

assessments. In using such data, panels will recognise the limited value of citation data for

recently published outputs, the variable citation patterns for different fields of research, the

possibility of ‘negative citations’, and the limitations of such data for outputs in languages

other than English. Panels will also be instructed to have due regard to the potential equality

implications of using citation data as additional information, as set out in ‘Equality briefing

for REF panels’ (REF 2018/05).

290.

Where sub-panels make use of citation data, it will be made available to them

as follows:

The REF team will procure a single source of citation data that provides a good level

of coverage across all UOAs in which the sub-panels will make use of such data.

Outputs entered onto the REF submission system by HEIs will be matched by the

REF team and/or our contractors against this database, using DOIs and other

bibliographic data entered onto the submissions system by HEIs. Institutions will

REF 2021 67

be able to verify these matches through the submission system, and to view the

citation counts in the same form that they will be provided to panels (although we

will continue to count citations made after the submission deadline, to provide

panels with up-to-date information).

For all matched outputs submitted by HEIs in the relevant UOAs, the REF team and/or

our contractor will provide REF panels with a count of the number of times the output

has been cited up to a fi point in time, at the start of the assessment phase.

291.

All sub-panels that make use of citation data in the assessment will have access to the

data provided on a consistent and transparent basis; submissions may not include details of

citations within any statements of additional information for outputs.

292.

Those panels that use citation information will continue to rely on expert review as

the primary means of assessment. The funding bodies do not sanction or recommend that

HEIs rely on citation information to inform the selection of outputs for inclusion in their

submissions. Institutions should select and submit outputs that in their judgement reflect

their highest-quality research in relation to the full range of assessment criteria (‘originality,

significance and rigour’), and in accordance with their codes of practice (see REF 2019/03),

having due regard to the equality implications of using citation data

Access to submitted outputs

293.

The REF team will attempt to source all submitted journal articles and conference

proceedings in electronic format directly from the publishers. We will therefore require the

submission of a DOI number wherever possible for these types of output.

294.

For all other output types, and where we are unable to source journal articles

and conference proceedings from the publishers, we will require institutions to make

available either:

the output in electronic format, wherever available, for example as DOI, or other

URL, or a PDF. Information on acceptable formats can be found in Annex K: Output

glossary and collection formats.

if not available in electronic format, a physical copy of the output or appropriate

evidence of the output. This may include digital/electronic material on a media

storage device, for example USB.

295.

Outputs may include multiple items that represent one output. In these cases, all

the items in a given output must be provided either in electronic format or deposited as a

physical output; not as a mixture of the two. For example, institutions should not provide

some of the output as a PDF upload and then send another part of the same output as a

‘physical’ output or ‘hard’ copy.

296.

Further details of the method of submission will accompany the pilot version of the

submission system software in autumn 2019.

16.

‘Equality briefing for REF panels’ (REF 2018/05), available at

www.ref.ac.uk.

68 REF 2021

Part 3 Section 3: Impact (REF3)

Definition of impact for the REF

297.

For the purposes of the REF, impact is defined as an effect on, change or benefit to the

economy, society, culture, public policy or services, health, the environment or quality of life,

beyond academia.

298.

Impact

includes

, but is not limited to, an effect on, change or benefit to:

•

the activity, attitude, awareness, behaviour, capacity, opportunity, performance,

policy, practice, process or understanding

•

of an audience, beneficiary, community, constituency, organisation or individuals

•

in any geographic location whether locally, regionally, nationally or internationally.

299.

Impact includes the reduction or prevention of harm, risk, cost or other negative

effects.

300.

For the purposes of the impact element of the REF:

Academic impacts on research or the advancement of academic knowledge

(whether in the UK or internationally) are excluded. (The submitted unit’s

contribution to academic research and knowledge is assessed within the ‘outputs’

and ‘environment’ elements of REF.)

Impacts on students, teaching or other activities both within and/or beyond the

submitting HEI are included. The ‘Panel criteria’ (paragraphs 301 to 302) sets out the

panels’ expectations for impact in this area.

301.

Impacts will be assessed in terms of their ‘reach and significance’ regardless of the

geographic location in which they occurred, whether locally, regionally, nationally or

internationally. The UK funding bodies expect that many impacts will contribute to the

economy, society and culture within the UK, but equally value the international contribution

of UK research.

302.

The ‘Panel criteria’ provides further guidance in relation to how the panels will assess

the case studies against the criteria of reach and significance and the kinds of impact that

the panels would anticipate from research across the UOAs; this guidance is not restrictive,

and any impact that meets the general definition in Annex C will be eligible.

Submission requirements for impact

303.

The REF aims to assess the impact of excellent research undertaken within each

submitted unit. This will be evidenced by specific examples of impacts that have been

underpinned by research undertaken within the unit over a period of time. The focus of the

assessment is the impact of the submitted unit’s research, not the impact of individuals or

individual research outputs, although they may contribute to the evidence of the submitted

unit’s impact.

304. Each submission must include impact case studies (REF3) describing specific impacts

that have occurred during the assessment period (1 August 2013 to 31 July 2020) that were

REF 2021 69

underpinned by excellent research undertaken in the submitted unit. The underpinning

research must have been produced by the submitting HEI during the period 1 January 2000

to 31 December 2020

305.

Panels will assess all the evidence provided in the submitted case studies (REF3),

and will form an impact sub-profile for each submission. Panels will apply their expert

judgement based on all the information provided in the impact case studies, before

confirming the impact sub-profiles.

306.

When writing case studies, submitting units should refer to the guidelines for

presenting quantitative data set out in the ‘Guidelines for standardising quantitative

indicators of impact within REF case studies’ (available at www.ref.ac.uk under Guidance).

These guidelines have been developed to enable more consistent presentation of

quantitative evidence in case studies, both to better inform the panels’ assessment and to

enable more effective analysis of the case studies post-REF 2021 by the funding bodies and

other stakeholders.

Impact case studies that include confidential information

307.

The following arrangements are in place to enable institutions to submit case studies

that include confidential information, with the agreement of the relevant organisation(s):

All panel members, assessors, observers and the panel secretariat are bound

by confidentiality arrangements. The current confidentiality and data security

arrangements are included in the ‘Panel criteria’. Panel members’ obligations during

the assessment phase will be expanded on, to include specific arrangements for

their treatment of confidential or sensitive information in submissions. These

expanded arrangements will be published in advance of the submission deadline.

Where there are main or sub-panel members or assessors who HEIs believe

would have a conflict of interest in assessing specific case studies, HEIs can identify

these when making submissions, and the case studies will not be made available

to such individuals.

When making submissions, HEIs can identify specific case studies that either

should not be published at all due to their confidential nature, or that should be

redacted prior to publication. HEIs will need to provide redacted versions suitable

for publication by 29 January 2021. Submitted case studies identified as ‘not for

publication’ or the elements for ‘redaction’ will be destroyed by the REF team once

no longer required for assessment purposes.

To protect panel members from potentially inappropriate exposure to intellectual

property, sub-panel chairs may identify specific panel members who should not

have access to, or should have access only to the redacted versions of, specific case

studies that include commercially sensitive information.

17.

The end of the period for the underpinning research (31 December 2020) extends beyond the end of the

period for the impact (31 July 2020). This is to align with the end of the publication period for outputs, and

recognises that research may have had impact prior to the publication of the outputs.

70 REF 2021

308.

In addition to the general arrangements set out in paragraph 307 above, there may be

specific instances where research has had impacts of a sensitive nature where the material

to be included in a case study could only be made available for assessment to individuals

with national security vetting clearance. This may relate to the underpinning research, the

nature of the impact, or both. The following arrangements apply, to enable the submission

of such specific cases:

The submitting HEI must request advance permission from the REF director to

submit such case studies, by providing outline information about the broad nature

of the research and/or impact, the level of sensitivity of the intended material, and

the level of clearance required of individuals to whom the full case study could

be made available. There will be three staggered deadlines for requests in May,

September and December 2019.

Permission will be granted to submit such case studies where the REF director

considers, having consulted the relevant panel chairs, that:

the confidentiality arrangements outlined in paragraph 307 above are

insufficient to enable the institution to submit the case study in the normal way

for assessment by the panel

and

ii.

it is practicable to identify existing panellists or appoint additional assessors who

have the appropriate clearance and expertise, and do not have direct conflicts of

interest, to assess the material.

Where permission is granted, arrangements will be made for the HEI to make the

case study available securely to the appropriate panel members/assessors. Only

the outline information will be made available to the panel and no details about

these case studies will be published.

HEIs should allow sufficient time for such case studies to go through the relevant

organisation’s internal release processes.

Number of case studies in a submission

309.

The number of case studies required in each submission will be determined by the

number (FTE) of Category A submitted staff returned in the submission, as set out in Table

1. If a submission includes fewer than the required number of case studies, a grade of

unclassified will be awarded to each required case study that is not submitted. Submissions

may not include more than the required number of case studies.

REF 2021 71

Table 1: Number of case studies required in submissions

Number of Category A submitted staff

submitted (FTE)

Required number of case studies

Up to 19.99

20 to 34.99

35 to 49.99

50 to 64.99

65 to 79.99

80 to 94.99

95 to 109.99

110 to 159.99

160 or more

10, plus one further case study per

additional 50 FTE

310.

Submissions will

not

be expected to provide impact case studies that are

representative of the spread of research activity across the whole submitted unit.

Institutions should select the strongest examples of impact that are underpinned by the

submitted unit’s excellent research, and should explain within the environment template

(REF5b) how the selected case studies relate to the submitted unit’s approach to enabling

impact from its research.

Eligibility definitions for case studies

311.

Each case study must provide details of a specific impact or impacts that:

meets the definition of impact for the REF in Annex C

occurred during the period 1 August 2013 to 31 July 2020 (see paragraph 312)

was underpinned by excellent research produced by the submitting unit in the

period 1 January 2000 to 31 December 2020 (see paragraphs 318 to 320).

312.

Case studies must describe impacts that occurred specifically within the period

1 August 2013 to 31 July 2020. The impacts may have been at any stage of development or

maturity during this period, so long as some effect, change or benefit meeting the definition

of impact in Annex C took place during that period. This may include, for example, impacts

at an early stage, or impacts that may have started prior to 1 August 2013 but continued

into the period 1 August 2013 to 31 July 2020. Case studies will be assessed in terms of the

reach and significance of the impact that occurred only during the period 1 August 2013

to 31 July 2020, and not in terms of any impact prior to this period or potential future or

anticipated impact after this period.

72 REF 2021

313.

More than one submitted unit (within the same HEI and/or in different HEIs) may

include the same impact within their respective case studies, so long as each submitted unit

produced excellent research that made a distinct and material contribution to the impact.

In such cases, units may provide common descriptions of the impact arising, where they

so wish.

Impact case studies continued from REF 2014

314.

All impact case studies submitted in REF 2021 must meet the same eligibility criteria,

including the length of the window for underpinning research and the assessment period

for the impact described (see paragraph 311). Where they meet these eligibility criteria,

case studies continued from examples submitted in 2014 will be eligible for submission in

REF 2021

315.

Submitting units will be required to identify continued case studies in the case study

template. This information will be made available to sub-panels and will be used by the

funding bodies in post-assessment evaluations. The ‘Panel criteria’ (paragraphs 292 to 295)

sets out further information about the main panels’ expectations in relation to receiving

continued case studies.

316.

Case studies will be considered to be continued if

both

the body of underpinning research is the same as described in a 2014 case study.

This should not be understood solely in relation to the referenced outputs, but

means that the continued case study does not describe any new research having

taken place since the previous case study that has made a distinct and material

contribution to the impact

and

there is significant overlap in the impact described, so that the impact types and

beneficiaries are broadly the same as described in the 2014 case study.

317.

A case study will be considered new where additional underpinning research has

taken place since that described in the previous case study, which has made a distinct and

material contribution to the impact, and/or the impact types or beneficiaries have changed.

Underpinning research

318.

To be eligible for assessment as an impact, the impact described in a case study must

have been underpinned by excellent research produced by the submitting unit, during the

period 1 January 2000 to 31 December 2020 (see footnote 17). Underpinning research may

be a body of work produced over a number of years or may be the output(s) of a particular

project. It may be produced by one or more individuals.

319.

Each case study must describe the underpinning research, include references to one

or more key research outputs, provide evidence of the quality of that research, and explain

how that research underpinned or contributed to the impact. Further guidance on the

information required in case studies is in Annex G. The following definitions apply:

‘Research produced by the submitting unit in the period 1 January 2000 to 31

December 2020’ means that staff carried out research within the scope of the

relevant UOA descriptor, while working in the submitting HEI (even if those staff

REF 2021 73

have since left). This research must be evidenced by outputs referenced in the case

study, first made publicly available between 1 January 2000 and 31 December 2020.

The staff may, but need not, have been submitted to a previous RAE/REF or to the

current REF. The research outputs may, but need not, have been submitted to a

previous RAE/REF or to the current REF.

i. Research by staff ‘working in the submitting HEI’ may include research

undertaken by staff who would be considered ‘Category C’ (defined in

paragraphs 375 to 376), on the basis that their research was clearly focused in

the submitting HEI. It may include research undertaken by staff employed on

non-Category A eligible contracts. The individuals need not be working in the

submitting HEI on the census date but must have been at the time they carried

out the underpinning research.

ii. Research undertaken solely by research students is not considered as having

been carried out by staff while working in the submitting HEI.

iii.

Provided the underpinning research is within the scope of the UOA in which the

case study is submitted, a case study may be submitted in a different UOA from

the individual who carried out the underpinning research.

iv.

If staff employed by the submitting HEI on the census date conducted all of the

research underpinning an impact before joining the institution, the submitting

HEI may not submit the impact of this research. (In this case, the institution

where the staff conducted the research may submit the impact.)

‘Excellent research’ means that the quality of the research is at least equivalent

to two star: ‘quality that is recognised internationally in terms of originality,

significance and rigour’. Each case study must include references to one or more

research outputs that best illustrate the research underpinning the impact and

were produced by the submitting HEI, and evidence of the quality of the research

as requested in the ‘Panel criteria’. Underpinning research outputs may include the

full range of output types listed in the output glossary (Annex K) and are not limited

to printed academic work. Panels will consider the evidence of research quality,

and may review outputs referenced in a case study. A panel will grade a case study

as unclassified if it judges that the underpinning research as a whole was not of at

least two-star quality.

‘Underpinned by’ means that the research made a distinct and material

contribution to the impact taking place, such that the impact would not have

occurred or would have been significantly reduced without the contribution of

that research. The relationship between research and impact can be indirect or

non-linear (for example, co-produced research). Each case study must explain how

(through what means) the research led to or contributed to the impact, and include

appropriate sources of information external to the HEI to corroborate these claims

(see Annex G). Where the panel judges that the submitted unit’s research did not

make a distinct and material contribution to the impact, the case study will be

graded as unclassified.

74 REF 2021

320.

Where a submitting HEI is the result of a merger between former HEIs, the submitting

HEI can submit impacts from the research undertaken by the former, now merged, HEIs.

321.

Where a submitting HEI has taken over a research unit – whether from another

HEI or from elsewhere – the submitting HEI can submit impacts from research that was

undertaken by the absorbed unit before it became part of the submitting HEI, with prior

agreement from the relevant UK funding body.

322.

Prior agreement must be sought by providing details of the nature of the research unit

and of when and how it became part of the submitting HEI, to info@ref.ac.uk, no later than

30 June 2020. The REF team will liaise with the relevant funding body and communicate the

decision to the HEI.

323.

In each case, the funding bodies will take into consideration whether a distinct unit

was absorbed by the submitting HEI in its entirety, and the extent to which there has been

genuine structural change.

324.

For clarity, these arrangements do not apply to impacts from research carried out by

individuals before they joined the submitting HEI. See paragraph 319.a.iv.

325.

There are many ways in which research may have underpinned impact, including but

not limited to:

Research that contributed directly or indirectly to an impact. For example, a

submitted unit’s research may have informed research in another submitted unit

(whether in the same or another HEI), which in turn led to an impact. In this case,

both submitted units may show that their research made a distinct and material

contribution to the impact.

Bodies of work produced over a number of years, or in the output(s) of a particular

project, conducted by one or more individuals, teams or groups, within one or

more submitted units that led to or underpinned an impact. More than one

submitted unit (within the same HEI or in different HEIs) may include the same

impact within their respective case studies, so long as each submitted unit

produced excellent research that made a distinct and material contribution to the

impact.

Impacts on, for example, public awareness, attitudes, understanding or behaviour

that arose from engaging the public with research. In these cases, the submitting

unit must show that the engagement activity was, at least in part, based on the

submitted unit’s research and drew materially and distinctly upon it. Further

guidance and examples are set out in the ‘Panel criteria’, Annex A.

Researchers that impacted on others through the provision of professional advice

or expert testimony. In such a case, the submitting unit must show that the

researcher’s appointment to their advisory role, or the specific advice given, was

at least in part based on the submitted unit’s research and drew materially and

distinctly upon it.

REF 2021 75

Research that led to impact through its deliberate exploitation by the HEI or

through its exploitation by others. The submitting HEI need not have been involved

in exploiting the research, but must show that its research made a distinct and

material contribution to the impact.

326.

Institutions must submit impact case studies in the appropriate UOAs. Impacts from

research undertaken at the submitting HEI may be submitted either in the REF UOA that

relates to the underpinning research, or, if this differs, to the REF UOA that relates to the

staff who conducted the research.

Case study data requirements (form REF3)

327.

Submitting units are required to submit case studies using a generic template.

The template, annotated with guidance, is in Annex G. The template has been developed

following REF 2014 with the addition of the following required fields to enable submitting

units in all UOAs to provide key information about the eligibility of the case study:

•

institution

•

unit of assessment

•

title of case study

•

period when the underpinning research was undertaken

(within the eligible timeframe)

•

names and roles (for example, job titles) of staff conducting the underpinning

research from the submitting unit (‘role’ at time when the underpinning

research was conducted)

•

period when staff involved in the underpinning research were employed by

the submitting HEI

•

period when the claimed impact occurred

•

whether the case study is continued from a case study submitted in 2014.

328.

The remaining sections of the template will allow HEIs to clearly explain and

demonstrate the impact of their research through a narrative that includes indicators and

evidence as appropriate to the case being made, and in a format that is suitable for panels

to assess them.

329.

In addition, submitting units are required to complete, where applicable, the following

additional contextual data fields. This information will enable research funders to track and

evaluate the impact of their funding. It will not be provided to the panels and will not form

part of the five-page limit for impact case studies:

•

name(s) of funder(s)

•

Global Research Identifier of funder(s): https://www.grid.ac/

•

name(s) of funding programme(s)

•

grant number(s)

•

amount of grant (in GBP (Sterling))

•

ORCID for each named researcher, where held

•

name(s) of formal partner(s)

•

country/countries where the impact occurred.

76 REF 2021

The information provided in these fields will facilitate the use and analysis of case studies

following the end of the exercise, rather than in the assessment process itself, and the data

will not be routinely provided to the panels.

330.

Institutions are required to provide to the REF team the corroborating evidence for

submitted impact case studies by 29 January 2021.

331.

We will collect, store and process all personal data submitted by HEIs to the REF in

accordance with current data protection legislation – the General Data Protection Regulation

(EU) 2016/679 and the Data Protection Act 2018. The evidence will be held by the REF

team on the secure submission system and will not be routinely provided to the sub-

panels. Where requested, information will be shared via a secure system with panel chairs,

members, assessors, panel secretariat and observers, who are all bound by confidentiality

arrangements. The information will be used to corroborate the claims made in the impact

case studies and will not be anonymised. Personal data will be retained until the end of the

assessment period and will be destroyed in December 2021.

332.

The onus is on submitting units to provide appropriate evidence within each case study

of the particular impact claimed. The REF panels provide guidance in the ‘Panel criteria’

about the kinds of evidence and indicators of impact they would consider appropriate to

research in their respective UOAs, but this guidance is not exhaustive.

333.

HEIs may submit corroborating evidence in any language. If the corroborating evidence

is a pre-existing document not available in English, the HEI should return the document in

its original language and state what language it is in. The REF team will use the expertise of

specialist advisers with the relevant language skills, if corroboration through these sources

is required.

334.

Corroborating contacts should be given only for people who the REF team can

communicate with in English.

335.

The information provided in an impact case study may be presented in any form

the institution considers to be appropriate. This may include tables and non-text content

(for example, diagrams, images), so long as the guidance on maximum page limits and

minimum font size, line spacing and margin widths are adhered to.

336.

Institutions may include URLs in REF3 only for the purpose of verifying or corroborating

claims made in the submission. Panels will not follow URLs to access additional evidence or

information to supplement the submission.

REF 2021 77

Part 3 Section 4: Environment data (REF4a/b/c)

337.

The REF sub-panels will create an environment sub-profile for each submission by

assessing the information submitted in REF5b (the environment template), taking into

account the information in REF5a (the institutional-level statement), and informed by the

data submitted in REF4a/b/c, as described in this section.

338.

When submissions are provided to the sub-panels, a standard analysis of the

quantitative data submitted in REF4a/b/c, in respect of each submission in that UOA, and

aggregated for all submissions in that UOA, will be provided as listed in Annex

Panels will

consider these data within the context of the information provided in REF5b, and within the

context of the disciplines concerned. The ‘Panel criteria’ indicates how the data analyses will

be used in informing the assessment of the research environment.

Research doctoral degrees awarded (REF4a)

339.

Each submission must include the number of research doctoral degrees awarded

each academic year (1 August to 31 July) 2013–14 to 2019–20 to students supervised within

the submitted unit.

340.

The REF team will provide to institutions data collected by HESA on the numbers of

research doctoral degrees awarded, to help in preparing submissions. We will provide data

collected by HESA for academic years (1 August to 31 July) 2013–14, 2014–15, 2015–16,

2016–17, 2017–18 and 2018–19, at both the level of academic cost centre, and by REF 2014

UOA (based on the supervisors’ REF 2014 UOA). The REF team will not provide data for

2019–20, and institutions will be required to complete the data for that year based on their

own data sources. These figures will be verified by the REF team following submission when

the HESA data are available.

341.

In preparing their submissions, institutions should allocate these data to the relevant

REF UOAs they are submitting in; or they may prepare their data from internal systems

using HESA definitions. In either case, the REF submission system will limit the extent to

which the total number of doctoral degrees awarded that is submitted by the institution as

a whole can exceed the total reported by the institution to HESA in each academic year and

across the period 2013–14 to 2018–19.

342.

The REF team will compare the data submitted to the REF with the HESA data and this

will inform our selection of submissions to be audited. If audited, an institution will need to

explain any significant variances with the totals submitted to HESA, and to show how they

have allocated data to the appropriate UOAs in their REF submissions.

Formal collaborative programmes for research training

343.

Data on formal collaborative programmes for research training was captured for

the first time on the 2015–16 HESA Student Record. This formally recorded individual

contributions made by HEIs that participated in collaborative programmes, and will enable

18.

These are students returned in the HESA Student Record whose qualification awarded is recorded as

‘Doctorate degree obtained primarily through advanced supervised research written up as a thesis/

dissertation’ or as a ‘New Route PhD’ (identified as codes ‘D00’ and ‘D01’ respectively in the QUAL field) for

the 2013–14 to 2016–17 records, and ‘Doctorate degree that meets the criteria for a research-based higher

degree’ (currently identified as code ‘D00’ in the QUAL field).

78 REF 2021

the individual HEI contributions to be reflected for 2015–16 onwards in the environment

data presented to the 2021 REF assessment panels.

344.

For both ‘concurrent supervision’ and ‘sequential supervision’ (described in paragraph

345), we will link recent years of student data (where this is appropriate) and apportion each

degree awarded to collaborating institutions in proportion to the student FTE supervised

by each collaborating institution. Within this process, any student FTE supervised by an

external organisation will be attributed to the reporting HEI.

345.

‘Concurrent supervision’ is where several institutions supervise the student

concurrently and one institution has been nominated by the others as the reporting

institution. The reporting institution records all the data for the student when completing

the HESA Student Record, identifying separately all the supervision undertaken by the other

institutions (or organisations). ‘Sequential supervision’ is where the student first studies at

one institution and then moves to a second institution, with the collaborating institutions

agreeing a formal handover of the student. In such cases, one institution stops recording

the student in the HESA Student Record and another institution begins recording the

student from the handover date.

Research income (REF4b)

346.

Each submission must include data on the submitted unit’s external research income

for each academic year 2013–14 to 2019–20, according to HESA definitions of research

income in the Finance Record, Table 5, broken down by source of income.

347.

For submissions in all UOAs in Main Panel A (UOAs 1–6), research income awarded

through open competition from the following bodies should be excluded from the source

‘UK central government bodies’ and returned in a separate line:

•

the National Institute for Health Research (NIHR)

•

the Chief Scientist Office, Scottish Government Health and Social Care Directorate

•

Health and Care Research Wales (formerly the National Institute for Social Care

and Health Research)

•

Health and Social Care Research & Development (HSC R&D), Northern Ireland.

These data will be reported to panels alongside income from the source ‘BEIS (Department

for Business, Energy and Industrial Strategy) Research Councils’, reflecting the competitive

nature of such income.

348.

The REF team will provide to institutions research income data collected by HESA in

the Finance Record Table 5 for academic years (1 August to 31 July) 2013–14, 2014–15,

2015–16, 2016–17, 2017–18 and 2018–19, at the level of academic cost centre, by source of

income. For 2019–20, the REF team will not provide the data and institutions will be required

to complete the data for this year based on their own data sources. These figures will be

verified by the REF team following submission when the HESA data are available.

REF 2021 79

349.

Data on research income for 2015–16 to 2019–20 will be presented to assessment

panels as an average over five years in order to moderate the more dynamic effects arising

from the introduction of the Financial Reporting Standard (FRS) 102 in 2015–16. One of

the key implications of FRS 102 is that a significant grant for an asset (such as a building or

equipment) may now appear in full in income in a particular year, instead of being reported

in fractional shares spread over the expected lifetime of the asset. This could result in more

significant year-on-year fluctuations in income than previously. Presenting assessment

panels with research income averaged over a five-year period helps to moderate these

effects. In reporting this income, we will include the balance of research-related deferred

capital and revenue grants as at 31 July 2015 taken to reserves that would otherwise not be

included in institutions’ accounts in any year because of the introduction of FRS 102. (This is

income that would have been spread over future years under the old accounting method,

but would have been counted in full in an earlier year under the new method

350.

In preparing their submissions, institutions should allocate these data to the relevant

UOAs they are submitting in; or they may prepare their data from internal systems using the

same definitions as in the HESA Finance Record Table 5. In either case, the REF submission

system will limit the extent to which the total income submitted by the institution to the REF

can exceed the totals reported by the institution to HESA in each year (total for all sources)

and across the assessment period as a whole, by each source. The REF team will compare

the data submitted to the REF with the HESA data, and this will inform our selection of

submissions to be audited. If audited, an institution will need to explain any significant

variances with the totals submitted to HESA, and to show how they have allocated data to

the appropriate REF UOAs.

Other notes

351.

We will exclude research income that is passed on to other institutions or organisations

as part of a collaborative project or subcontracted work to the extent that this is identified

separately in the 2013-14 and 2014-15 HESA data.

352.

Where a grant or contract is held, or work conducted, across more than one UOA it

should be divided between submissions in different UOAs according to the way the income

has been used. Research projects which are funded from several sources should have their

income allocated under the respective headings to reflect the actual source of the income.

Research income-in-kind (REF4c)

353.

The estimated value of Research Council facility time allocated through peer review

and used by researchers at submitted units will be provided to institutions for use in

preparing submissions and should be returned in REF4c. For submissions in UOAs 1–6, the

estimated value of equivalent income-in-kind from the health research funding bodies listed

in paragraph 347 will also be provided.

354. Only income-in-kind which has not been reported as research income to the HESA

Finance Record should be reported as income-in-kind to the REF.

19.

HEFCW has provided to the REF team data that has been collected from Welsh HEIs on deferred capital and

revenue grants for 2015–16, for research which would otherwise have been omitted from the HESA record

because of the introduction of FRS 102. This allows HEIs to include these data in their REF submissions. The

data were collected by HESA for HEIs in England, Scotland and Northern Ireland.

80 REF 2021

355.

The lead Research Council responsible for access and funding of a named facility will

supply data on the value of these allocations to the HEIs concerned and to the REF team.

We expect that the Research Councils and the bodies listed in paragraph 347 will supply

data relating to the period 1 August 2013 to 31 July 2020, and these will be made available

to HEIs in sufficient time to be included in their submissions. We are consulting with these

bodies on the arrangements and will issue further guidance in spring 2019, including

confirmation of the provisional dates for the supply of data to HEIs and a list of facilities.

356.

The income-in-kind data will be reported to panels alongside research income from

‘BEIS Research Councils’ and will be identified in a separate line.

357.

The REF team will compare the data on income-in-kind provided by the Research

Councils and the bodies listed in paragraph 347 with those submitted to the REF, and

this will inform our selection of submissions to be audited. If audited, an institution will

need to explain any significant variances and to show how they have allocated data to the

appropriate REF UOAs.

358.

Some REF panels may, by exception, state in their criteria documents that they require

data about other specific sources of income-in-kind. Where this is the case, submissions

should include such data within the relevant section of the environment template (REF5b)

and not on REF4c.

Part 3 Section 5: Environment (REF5a/b)

Submission requirements for the institutional-level environment statement (REF5a)

359.

Information is required about the institution’s strategy and resources to support

research and enable impact, relating to the period 1 August 2013 to 31 July 2020.

A template is provided for REF5a in Annex H. Other than the exception set out in paragraph

364, one statement will be required for each submitting institution, consisting of the

following sections:

Context and mission: an overview of the size, structure and mission of

the institution.

Strategy: the institution’s strategy for research and enabling impact (including

integrity, open research, and structures to support interdisciplinary research)

in the assessment period and for the next five-year period.

People: the institution’s staffing strategy, support and training of research students,

and building on the information provided in codes of practice, evidence about how

equality and diversity in research careers is supported and promoted across the

institution.

Income, infrastructure and facilities: the institutional-level resources and

facilities available to support research, including mechanisms for supporting the

reproducibility of research as appropriate to the research focus of the HEI, and to

facilitate its impact.

REF 2021 81

360.

In providing evidence in the REF5a statement, institutions should draw on supporting

quantitative indicators where applicable. Institutions are encouraged to refer to the advice

and examples based on work carried out by the Forum for Responsible Research Metrics,

available at www.ref.ac.uk, under Guidance.

361.

Institutions may include URLs in REF5a only for the purpose of verifying or

corroborating claims made in the statement. Panels will not follow URLs to access additional

evidence or information to supplement the submission.

362.

The statement (REF5a) will be appended to each unit-level template (REF5b) submitted

for panel review. The sub-panels will take into account the information provided in the

institutional-level statement when assessing the unit-level template. The ‘Panel criteria’

describes how the sub-panels will use the information in form REF5a to inform the

assessment of the REF5b unit-level template. The institutional-level statement will

contribute to the assessment of the UOA-level statement but will not be separately scored

by the sub-panels.

363.

A pilot exercise on the standalone assessment of a discrete institutional-level

environment element will run concurrently to the REF 2021 assessment. The REF5a

statements will be reviewed by a pilot assessment panel for this purpose. The outcomes

from the pilot assessment will not be included in the REF 2021 outcomes, but will inform

the inclusion of a discrete institutional-level environment element in future research

assessment. Further information about the pilot exercise will be made available on the REF

website, www.ref.ac.uk.

364.

Small and specialist institutions that will make a submission in one UOA only will not

be required to provide a REF5a statement but may choose to submit one where this is the

most appropriate way of representing the institution’s research environment. Where HEIs

choose not to provide a REF5a statement, the pilot panel will review the submitted REF5b

template. In this case, institutions should ensure that sufficient information is provided in

the REF5b template, bearing in mind the guidance set out in the ‘Panel criteria’. Additionally,

where there is any distinction between the research and impact strategies, policies, facilities

and resources between the institution and the submitting unit, this should be clearly

identified in the REF5b template.

365.

The completed REF5a environment statement must be submitted according to the

guidance on formatting and word limits, set out in Annex F.

82 REF 2021

Submission requirements for the unit-level environment template (REF5b)

366.

Information is required about the environment for research and enabling impact for

each submitting unit, relating to the period 1 August 2013 to 31 July 2020. A template for

REF5b is provided in Annex I. Each submission must include a single completed REF5 form,

consisting of the following sections:

Unit context, research and impact strategy.

People, including:

–

staffing strategy and staff development

–

research students

–

equality and diversity.

Income, infrastructure and facilities.

Collaboration and contribution to the research base, economy and society.

367.

The information provided in REF5b should relate to the environment of the submitted

unit, and should not duplicate information about the institutional-level environment that is

provided in REF5a.

368.

Where a submission includes staff from distinct ‘departments’ or other organisational

units, the submission should explain this and any distinctive aspects of the research

environments of these organisational units, within each section of the environment

template. There is no expectation that the environment element of a submission will relate

to a single department or coherent organisational unit.

369.

In providing evidence in the REF5b template, institutions should draw on supporting

quantitative indicators where applicable. Detailed guidance on the requirements for the

content of REF5b is provided in the ‘Panel criteria’ (Part 2, Section 4). This includes guidance

on the use of quantitative indicators in REF5b, with reference to advice and examples based

on the work of the Forum for Responsible Research Metrics.

370.

Institutions may include URLs in REF5b, only for the purpose of verifying or

corroborating claims made in the submission. Panels will not follow URLs to access

additional evidence or information to supplement the submission.

371.

The ‘Panel criteria’ describes how the sub-panels will use the information in form

REF5b together with the data in forms REF4a/b/c in assessing the submissions to form the

environment sub-profiles. Each section of the environment template will be significant in

informing the environment sub-profile. The main panels have set out the weighting that will

be given in the assessment to each section of the template in the ‘Panel criteria’.

372.

To inform the sub-panels’ assessment of the ‘People’ section of the unit-level

environment template (REF5b), EDAP will provide advice on overall strengths or areas of

concern, on the basis of the generic assessment criteria and procedures. Each sub-panel will

retain responsibility for recommending the quality profile for all work that was submitted

in its UOA. EDAP will carry out a calibration exercise at an early stage in the assessment to

develop a common understanding of the assessment criteria and standards.

REF 2021 83

373.

Completed environment templates must be submitted according to the guidance on

formatting and word limits, set out in Annex F.

Category C staff

374.

Information about the contribution of Category C staff to the environment for research

and enabling impact may be provided in the ‘Collaboration and contribution’ section of the

REF5b template. The ‘Panel criteria’ sets out further guidance on which panels expect to

receive this information and the nature of the information requested.

375.

Category C staff are defined as individuals employed by an organisation other than an

HEI, whose contract or job role (as documented by their employer) includes the undertaking

of research, and whose research is primarily focused in the submitting unit.

376.

Category C staff may be employed by the NHS, a Research Council unit, a charity or

other organisation except for an HEI. For clarity, the following do not meet the definition of

Category C staff:

Any staff employed by the HEI, including vice-chancellors or heads of HEIs; HEI

staff on non-academic contracts, including those working in university museums

and libraries; or retired staff who are still active in research (where they satisfy the

definition in paragraph 120.h or, for retired staff, paragraph 120.b, these staff meet

the definition of Category A eligible staff).

Visiting professors, fellows and lecturers employed by other HEIs.

84 REF 2021

Annex A: Assessment criteria and level definitions

This annex sets out the generic criteria for assessing submissions and the definitions of

the starred levels in the overall quality profiles and each of the sub-profiles (for outputs,

impact and environment).

Sub-panels will use their professional judgement to form the overall quality profile to be

awarded to each submission, taking into account all the evidence presented. The primary

outcome of the assessment will be an overall quality profile awarded to each submission,

showing the proportion of the submission that meets each starred level in the profile.

The definitions of the starred levels in the overall quality profile are below.

Table A1: Overall quality profile: Definitions of starred levels

Four star

Quality that is world-leading in terms of originality, significance

and rigour.

Three star

Quality that is internationally excellent in terms of originality, significance

and rigour but which falls short of the highest standards of excellence.

Two star

Quality that is recognised internationally in terms of originality,

significance and rigour.

One star

Quality that is recognised nationally in terms of originality, significance

and rigour.

Unclassified

Quality that falls below the standard of nationally recognised work. Or

work which does not meet the published definition of research for the

purposes of this assessment.

For each of the three elements of the assessment – outputs, impact and environment –

sub-panels will develop a sub-profile, showing the proportion of the submission that meets

each of the four starred quality levels. The assessment criteria and the definitions of the

starred levels for the sub-profiles are set out below.

Table A2: Outputs sub-profile: Criteria and definitions of starred levels

The criteria for assessing the quality of outputs are ‘originality, significance and rigour’

Four star

Quality that is world-leading in terms of originality, significance

and rigour.

Three star

Quality that is internationally excellent in terms of originality, significance

and rigour but which falls short of the highest standards of excellence.

Two star

Quality that is recognised internationally in terms of originality,

significance and rigour.

One star

Quality that is recognised nationally in terms of originality, significance

and rigour.

Unclassified

Quality that falls below the standard of nationally recognised work. Or

work which does not meet the published definition of research for the

purposes of this assessment.

REF 2021 85

Table A3: Impact sub-profile: Criteria and definitions of starred levels

The criteria for assessing impacts are ‘reach’ and ‘significance’:

•

In assessing the impact described within a case study, the panel will form an overall

view about its ‘reach and significance’ taken as a whole, rather than assess ‘reach and

significance’ separately.

Four star

Outstanding impacts in terms of their reach and significance.

Three star

Very considerable impacts in terms of their reach and significance.

Two star

Considerable impacts in terms of their reach and significance.

One star

Recognised but modest impacts in terms of their reach and significance.

Unclassified

The impact is of little or no reach and significance; or the impact was

not eligible; or the impact was not underpinned by excellent research

produced by the submitted unit.

Table A4: Environment sub-profile: Criteria and definitions of starred levels

The research environment will be assessed in terms of its ‘vitality and sustainability’.

Panels will consider both the ‘vitality and sustainability’ of the submitted unit,

including its approach to enabling impact from its research, and its contribution

to the ‘vitality and sustainability’ of the wider research base.

Four star

An environment that is conducive to producing research of world-leading

quality and enabling outstanding impact, in terms of its vitality

and sustainability.

Three star

An environment that is conducive to producing research of

internationally excellent quality and enabling very considerable impact,

in terms of its vitality and sustainability.

Two star

An environment that is conducive to producing research of

internationally recognised quality and enabling considerable impact,

in terms of its vitality and sustainability.

One star

An environment that is conducive to producing research of nationally

recognised quality and enabling recognised but modest impact, in terms

of its vitality and sustainability.

Unclassified

An environment that is not conducive to producing research of nationally

recognised quality or enabling impact of reach and significance.

The ‘Panel criteria’ explains in more detail how the sub-panels will apply the assessment

criteria and interpret the level definitions in developing the sub-profiles.

The method for combining the sub-profiles into the overall quality profile is explained in

Annex B.

86 REF 2021

Notes on the criteria and definitions of the starred levels

‘World-leading’ quality denotes an absolute standard of quality in each unit of

assessment.

‘World-leading’, ‘internationally’ and ‘nationally’ in this context refer to quality standards.

They do not refer to the nature or geographical scope of particular subjects, nor to the

locus of research nor its place of dissemination. For example, research which is focused

within one part of the UK might be of ‘world-leading’ standard. Equally, work with an

international focus might not be of ‘world-leading, internationally excellent or internationally

recognised’ standard.

The criterion of ‘reach’ for impacts does not refer specifically to a geographic scale. Sub-

panels will consider a number of dimensions to the ‘reach’ of impacts as appropriate to the

nature of the research and its impacts. For example, an impact located within one region

of the UK might be judged as ‘outstanding’ (graded as four star). Equally, an impact with

international reach might not be judged as ‘outstanding’, ‘very considerable’

or ‘considerable’.

10.

The profile for a submission that contains no research which meets the one-star

threshold will be 100 per cent unclassified. A submission that contains no research

(that is, no work that meets the definition of research for the REF) will not be awarded a

quality profile.

REF 2021 87

Annex B: Quality profiles

The overall quality profiles will show the proportion of research activity in a submission

judged to meet the definitions at each starred level. The overall quality profiles will be

published in steps of one per cent. Table B1 shows overall quality profiles for two

fictional universities.

Table B1: Sample overall quality profiles*

Unit of

assessment A

FTE Category A

submitted staff

Percentage of research activity in the submission

judged to meet the standard for:

four

star

three

star

two

star

one

star

unclassified

University X

50.45

University Y

65.2

* The figures are for fictional universities. They do not indicate expected proportions.

Sub-panels will produce the overall quality profiles by assessing three distinct elements

of the assessment – research outputs, impact and the environment – to produce a sub-

profile for each element. The three sub-profiles will be aggregated to form the overall

quality profile for the submission, with each element weighted as follows:

•

Outputs: 60 per cent

•

Impact: 25 per cent

•

Environment: 15 per cent.

The rounding methodology described in paragraphs 4 to 7 of this annex will be used to

produce the overall quality profiles. In recommending the overall quality profiles to the main

panels, sub-panels will confirm that, in their expert judgement, the overall quality profile is

a fair reflection of the research activity in that submission, and that their assessment has

taken account of all the different components of the submission.

88 REF 2021

15%

14.7 31.4

Outputs

40.3

60%

u/c

13.6 0

Figure B1

Building a quality profile: a worked example

Overall Quality Profile

Quality Level

u/c

% of research

Environment

u/c

Impact

u/c

Rounding

The sub-profiles will be combined using the weights in paragraph 2 of this annex. A

cumulative rounding process will then be applied to the combined profile, to produce an

overall quality profile. This methodology will ensure that the overall quality profile for any

submission will always sum to 100 per cent and to avoid the unfair consequences that

simple rounding can produce.

Worked example

Using the example in Figure B1, first calculate the initial overall profile, that is, the sum

of the weighted sub-profiles for outputs, environment and impact.

Starred levels

u/c

Outputs

14.7

31.4

40.3

13.6

Environment

Impact

25%

REF 2021 89

Weighted

60%

8.82

18.84

24.18

8.16

15%

1.5

6.0

1.5

25%

5.0

10.0

8.75

1.25

Initial profile

15.3

34.8

38.9

10.9

Cumulative rounding works in three stages:

The initial profile is:

u/c

15.3

34.8

38.9

10.9

Stage 1: Calculate the cumulative totals (for example, the cumulative total at 3* or

better is 15.3 + 34.8 = 50.1).

3* or

2* or

1* or

u/c or

better

15.3

50.1

89.1

100

Stage 2: Round these to the nearest one per cent (rounding up if the percentage ends

in exactly 0.5).

3* or

2* or

1* or

u/c or

better

100

Stage 3: Find the differences between successive cells to give the rounded profile. So,

for example, the percentage allocated to 2* is the difference between the cumulative

total at 2* or better, minus the cumulative total at 3* or better (89 - 50 = 39).

u/c

Cumulating the totals the other way (rounding down if the percentage ends in exactly

0.5) gives exactly the same answer.

90 REF 2021

Annex C: Definitions of research and impact for the REF

Definition of research for the REF

For the purposes of the REF, research is defined as a process of investigation leading to

new insights, effectively shared.

includes

work of direct relevance to the needs of commerce, industry, culture,

society, and to the public and voluntary sectors; scholarship

; the invention and generation

of ideas, images, performances, artefacts including design, where these lead to new

or substantially improved insights; and the use of existing knowledge in experimental

development to produce new or substantially improved materials, devices, products and

processes, including design and construction. It excludes routine testing and routine

analysis of materials, components and processes such as for the maintenance of national

standards, as distinct from the development of new analytical techniques. It also

excludes

the development of teaching materials that do not embody original research.

includes

research that is published, disseminated or made publicly available in the

form of assessable research outputs, and confidential reports (as defined in paragraph 261).

Definition of impact for the REF

For the purposes of the REF, impact is defined as an effect on, change or benefit to the

economy, society, culture, public policy or services, health, the environment or quality of life,

beyond academia (as set out in paragraph 7).

Impact

includes

, but is not limited to, an effect on, change or benefit to:

•

the activity, attitude, awareness, behaviour, capacity, opportunity, performance,

policy, practice, process or understanding

•

of an audience, beneficiary, community, constituency, organisation or individuals

•

in any geographic location whether locally, regionally, nationally or internationally.

Impact

includes

the reduction or prevention of harm, risk, cost or other negative

effects.

For the purposes of the impact element of the REF:

Impacts on research or the advancement of academic knowledge within the HE

sector (whether in the UK or internationally) are

excluded

. (The submitted unit’s

contribution to academic research and knowledge is assessed within the ‘outputs’

and ‘environment’ elements of REF.)

Impacts on students, teaching or other activities both within and beyond the

submitting HEI are

included

(see the ‘Panel criteria’, paragraphs 301 to 302).

20.

Scholarship for the REF is defined as the creation, development and maintenance of the intellectual

infrastructure of subjects and disciplines, in forms such as dictionaries, scholarly editions, catalogues and

contributions to major research databases.

REF 2021 91

Annex D: Units of assessment

Main

panel

Unit of assessment

Clinical Medicine

Public Health, Health Services and Primary Care

Allied Health Professions, Dentistry, Nursing and Pharmacy

Psychology, Psychiatry and Neuroscience

Biological Sciences

Agriculture, Food and Veterinary Sciences

Earth Systems and Environmental Sciences

Chemistry

Physics

Mathematical Sciences

Computer Science and Informatics

Engineering

Architecture, Built Environment and Planning

Geography and Environmental Studies

Archaeology

Economics and Econometrics

Business and Management Studies

Law

Politics and International Studies

Social Work and Social Policy

Sociology

Anthropology and Development Studies

Education

Sport and Exercise Sciences, Leisure and Tourism

Area Studies

Modern Languages and Linguistics

English Language and Literature

History

Classics

Philosophy

Theology and Religious Studies

Art and Design: History, Practice and Theory

Music, Drama, Dance, Performing Arts, Film and Screen Studies

Communication, Cultural and Media Studies, Library and

Information Management

92 REF 2021

Annex E: Timetable

September 2017

Publication of ‘Initial decisions on the Research Excellence

Framework’ by the funding bodies, following consultation on

implementation of the Stern review recommendations (REF 2017/01)

October 2017

Publication of ‘Roles and recruitment of expert panels’ (REF 2017/03)

November 2017

Publication of ’Decisions on staff and outputs’ (2017/04)

March 2018

Panel membership for criteria phase announced

End of July 2018

Publication of draft ‘Guidance on submissions’ and ‘Panel criteria’

for consultation

15 October 2018

Close of consultation on draft ‘Guidance on submissions’ and

‘Panel criteria’

January 2019

Publication of final ‘Guidance on submissions’, ’Panel criteria’,

and ‘Guidance on codes of practice’; appointment of additional

EDAP members

Spring/

summer 2019

Institutions intending to make submissions to the REF submit their

codes of practice; invitation to request multiple submissions, case

studies requiring security clearance, and exceptions to submission

for small units (staggered deadlines in May, September and

December 2019); beta versions of the submission system will be

available in both test and live environments for institutions to use

Autumn 2019

Pilot of the REF submission system; survey of submissions

intentions opens; proposed date for inviting reduction requests

for staff circumstances

December 2019

Survey of submissions intentions complete; final deadline for

requests for multiple submissions, case studies requiring security

clearance, and exceptions to submission for small units; publication

of approved codes of practice

Early 2020

Formal release of the submission systems and accompanying

technical guidance; invitation to HEIs to make submissions; invitation

to nominate panel members and assessors for the assessment

phase; deadline for staff circumstances requests

Mid 2020

Appointment of additional members and assessors to panels

31 July 2020

Census date for staff; end of assessment period (for research

impacts, the research environment, and data about research income

and research doctoral degrees awarded)

27 November 2020

Closing date for submissions

31 December 2020

End of publication period (cut-off point for publication of research

outputs, and for outputs underpinning impact case studies)

29 January 2021

Deadline for providing further details for outputs pending

publication; redacted versions of impact case studies; and

corroborating evidence held for impact case studies

Throughout 2021

Panels assess submissions

December 2021

Publication of outcomes

Spring 2022

Publication of submissions, panel overview reports and sub-profiles

REF 2021 93

Annex F: Format and page limits for textual parts of

submissions

Format

Templates for REF3, REF5a and REF5b will be provided to institutions in Word.

Completed templates and case studies must be submitted as PDF documents for the

assessment. A Word version of the templates and case studies will also be required.

PDF documents must be accessible to screen reading technology (rather than scanned

documents). Completed templates must adhere to the following:

•

Arial font, 11 point (minimum)

•

single line spacing (minimum)

•

2 cm margins (minimum).

Completed templates may include formatting (bold or underlined text, headings, lists,

and so on), tables and non-text content, so long as the guidance on maximum page/word

limits and on minimum font size, line spacing and margin widths set out in this annex are

adhered to.

Template lengths

Impact case study (REF3) length

All case studies will be limited to

five pages

, including all references. This limit excludes

the personal details of the corroborating sources listed in Section B5 of the case study

template and the ‘required (where applicable)’ fields. These items of information will be

collected in a separate form.

Completed impact case study templates may include formatting (bold or underlined

text, headings, lists, and so on), tables and non-text content, so long as the guidance on

the maximum page limit, and on minimum font size, line spacing and margin widths are

adhered to.

Institutional-level environment statement (REF5a)

The

maximum

word limit for the institutional environment statement (REF5a) will

depend on the total FTE of Category A submitted staff returned across the institution,

according to Table F1. Submitting institutions are reminded that this is an upper limit,

not a minimum requirement.

Table F1: Page limits for REF5a

Number of Category A

submitted staff returned by

institution (FTE)

Word limit for environment

statement (REF5a)

1 – 99.99

4,000

100 – 499.99

4,500

500 – 999.99

5,000

1000 or more

5,500

94 REF 2021

Unit-level environment template (REF5b)

The

maximum

word limit for the unit-level environment template (REF5b) will depend

on the total FTE of Category A submitted staff included in the submission, according to

Table F2. Submitting institutions are reminded that this is an upper limit, not a minimum

requirement.

Table F2: Word limits for REF5b

Number of Category A

submitted staff in the

submission (FTE)

Word limit for environment

template (REF5b)

1 – 19.99

8,000

20 – 29.99

8,800

30 – 39.99

9,600

40 – 49.99

10,400

50 – 69.99

11,200

70 or more

12,000, plus 800 further words

per additional 20 FTE

REF 2021 95

Annex G: Impact case study template and guidance

This annex provides the template for impact case studies, annotated with guidance

about the information required in each of its sections. This should be read alongside the

definitions and eligibility criteria for impact case studies in Part 3, Section 3 of the ’Guidance

on submissions’, and alongside the ‘Panel criteria’. The case study template for use in

preparing submissions will be provided in Word, along with templates for REF5a and REF5b,

on the REF submission system.

Each case study should include sufficiently clear and detailed information to enable

panels to make judgements based on the information it contains, without making

inferences, gathering additional material, following up references or relying on members’

prior knowledge. References to other sources of information will be used for verification

purposes only, not as a means for panels to gather further information to inform

judgements.

The information fields in Section A are mandatory and will be made available to panels.

The additional contextual data fields are mandatory, where applicable. They will be

entered separately and will

not

be routinely provided to panels. They will not count towards

the page limit.

Each completed case study template will be limited to

five pages

in length (see Annex

F). Within the annotated template below, indicative guidance is provided about the expected

maximum length limit of each section, but institutions will have flexibility to exceed these so

long as the case study as a whole remains no longer than

five pages

and the guidance on

formatting in Annex F is adhered to.

When presenting numeric data, submitting units are strongly encouraged to adhere to

the guidelines set out in the ‘Guidelines for standardising quantitative indicators of impact

within REF case studies’ (available at www.ref.ac.uk under Guidance). This will enable more

effective analysis of the data in post-assessment evaluations.

96 REF 2021

Section A

The fields in this section are mandatory.

Institution:

Unit of Assessment:

Title of case study:

Period when the underpinning research was undertaken:

Details of staff conducting the underpinning research from the submitting unit:

Name(s):

Role(s) (e.g. job title):

Period(s) employed by

submitting HEI:

Period when the claimed impact occurred:

Is this case study continued from a case study submitted in 2014?

Y/N

The definition of continued case studies is provided in the ‘Guidance on submissions’,

paragraph 316.

Section B

1. Summary of the impact (indicative maximum 100 words)

This section should briefly state what specific impact is being described in the case study

Underpinning research (indicative maximum 500 words) See paragraphs 318 to 326.

This section should outline the key research insights or findings that underpinned the

impact, and provide details of what research was undertaken, when, and by whom. This

research may be a body of work produced over a number of years or may be the output(s)

of a particular project. References to specific research outputs that embody the research

described in this section, and evidence of its quality, should be provided in the next

section (section B3).

Details of the following should be provided in this section:

•

The nature of the research insights or findings which relate to the impact

claimed in the case study.

•

An outline of what the underpinning research produced by the submitted unit

was (this may relate to one or more research outputs, projects or programmes).

•

Any relevant key contextual information about this area of research.

3. References to the research (indicative maximum of six references)

This section should provide references to key outputs from the research described in the

previous section, and evidence about the quality of the research. Underpinning research

outputs may include the full range of types listed in the output glossary (Annex K) and are

not limited to printed academic work. All forms of output cited as underpinning research

will be considered equitably, with no one type of output being preferred over others.

REF 2021 97

Include the following details for each cited output:

•

author(s)

•

title

•

year of publication

•

type of output and other relevant details required to identify the output (for

example, DOI, journal title and issue)

•

details to enable the panel to gain access to the output, if required (for example, a

DOI or other URL), or stating that the output is listed in REF2 or can be supplied by

the HEI on request.

All outputs cited in this section must be capable of being made available to panels. If they

are not available in the public domain or listed in REF2, the HEI must be able to provide

them if requested by the REF team.

Evidence of the quality of the research must also be provided in this section. Guidance on

this is provided in the ‘Panel criteria’.

Where panels request details of key research grants or end of grant reports, the following

should be provided:

•

who the grant was awarded to

•

the grant title

•

sponsor

•

period of the grant (with dates)

•

value of the grant.

Details of the impact (indicative maximum 750 words). The ‘Panel criteria’, Annex A,

Table 1 provides an illustrative list of evidence that could be provided.

This section should provide a narrative, with supporting evidence, to explain:

•

how the research underpinned (made a distinct and material contribution to)

the impact;

•

the nature and extent of the impact.

The following should be provided:

•

A clear explanation of the process or means through which the research led

to, underpinned or made a contribution to the impact (for example, how it was

disseminated, how it came to influence users or beneficiaries, or how it came to be

exploited, taken up or applied).

•

Where the submitted unit’s research was part of a wider body of research that

contributed to the impact (for example, where there has been research collaboration

with other institutions), the case study should specify the particular contribution of

the submitted unit’s research and acknowledge other key research contributions.

•

Details of the beneficiaries – who or what community, constituency or organisation

has benefitted, been affected or impacted on.

•

Details of the nature of the impact – how they have benefitted, been affected

or impacted on.

•

Evidence or indicators of the extent of the impact described, as appropriate to

the case being made.

•

Dates of when these impacts occurred.

98 REF 2021

Where the sources are individuals who could be contacted or have provided factual statements to the

HEI, the submitted case study should state only the organisation (and, if appropriate, the position) of the

individuals concerned, and which claim(s) they can corroborate. Their personal details (name, position,

contact details) must be entered separately on the REF submission system and not on REF3. Details of a

maximum of five individuals may be entered for each case study; these data will not be published as part of

the submission.

Additional contextual data

The fields in this section are mandatory, where applicable. The information will be

used in post-assessment evaluations and will

not

be routinely provided to panels.

This information should be provided in a separate web form and is not included in

the five-page limit.

Name(s) of funder(s):

Global Research Identifier of funder(s) (https://www.grid.ac/):

Name(s) of funding programme(s):

Grant number(s):

Amount of grant (in GBP):

ORCID for each named researcher, where held:

Name(s) of formal partner(s):

Country/countries where the impact occurred**:

** Where the impact occurred specifically within one country that is part of the UK (for example, Wales), this

country rather than ‘UK’ should be specified in the country/countries field.

Sources to corroborate the impact (indicative maximum of ten references)

This section should list sources external to the submitting HEI that could, if requested by

panels, provide corroboration of specific claims made in the case study. Sources provided

in this section should not be a substitute for providing clear evidence of impact in Section

B4; the information in this section will be used for audit purposes only.

This section should list sufficient sources that could corroborate key claims made about

the impact of the unit’s research. These could include, as appropriate to the case study,

the following external sources of corroboration (stating which claim each source provides

corroboration for):

•

Reports, reviews, web links or other documented sources of information

in the public domain.

•

Confidential reports or documents (if listed, these must be submitted to the

REF team by 29 January 2021).

•

Individual users/beneficiaries who could be contacted by the REF team

to corroborate claims*.

•

Factual statements already provided to the HEI by key users/beneficiaries,

that corroborate specific claims made in the case study (if listed, these must be

submitted to the REF team by 29 January 2021)*.

REF 2021 99

Annex H: Institutional-level environment statement (REF5a)

This annex provides the template for REF5a. The template for use in preparing

submissions will be provided in Word, along with templates for REF3 and REF5b, on the REF

submission system.

Information about the approach to assessing REF5a in REF 2021, as well as guidance on

completing the template, is available in Part 3, Section 5 of this ‘Guidance on submissions’

document, and in Part 3, Section 4 of the ‘Panel criteria’.

In providing evidence in the REF5a statement, institutions should draw on supporting

quantitative indicators where applicable. Institutions are encouraged to refer to the advice

and examples based on work carried out by the Forum for Responsible Research Metrics,

available at www.ref.ac.uk, under Guidance.

Each completed template must be submitted according to the guidance on formatting

and word limits set out in Annex F of this document.

Institutional-level environment statement (REF5a)

Institution:

Context and mission

An overview of the size, structure and mission of the institution.

Strategy

The institution’s strategy for research and enabling and actualising impact (including

integrity, open research, and structures to support interdisciplinary research) in the

assessment period and for the next five-year period.

People

The institution’s staffing strategy, support and training of research students and, building

on the information provided in codes of practice, evidence about how equality and

diversity in research careers is supported and promoted across the institution.

Income, infrastructure and facilities

The institutional-level resources and facilities available to support research and enable

and actualise impact.

100

REF 2021

Annex I: Unit-level environment template (REF5b)

This annex provides the template for REF5b. The template for use in preparing

submissions will be provided in Word, along with templates for REF3 and REF5a, on the REF

submission system.

Guidance on completing the template is available in Part 3, Section 5 of this ‘Guidance

on submissions’ document, and in Part 3, Section 4 of the ‘Panel criteria’.

In providing evidence in the REF5b template, institutions should draw on supporting

quantitative indicators where applicable. In identifying additional indicators for inclusion,

submitting units are strongly advised to refer to the advice and examples based on work

carried out by the Forum for Responsible Research Metrics, available at www.ref.ac.uk,

under Guidance. Further guidance is provided in Part 3, Section 4 of the ‘Panel criteria’.

Each completed template must be submitted according to the guidance on formatting

and word limits set out in Annex F of this document.

Unit-level environment template (REF5b)

Institution:

Unit of assessment:

Section 1. Unit context and structure, research and impact strategy

This section should provide evidence of the submitted unit’s achievement of strategic aims

for research and impact during the assessment period, and details of future strategic aims

and goals for research and impact; how these relate to the structure of the unit; and how

they will be taken forward

Section 2. People

This section should provide evidence about staffing strategy and staff development within

the submitted unit; support mechanisms for, and evidence of the training and supervision

of, PGR students; and evidence of how the submitting unit supports and promotes

equality and diversity.

Section 3. Income, infrastructure and facilities

This section should provide information about the submitted unit’s income, infrastructure

and facilities pertaining to research and research impact.

Section 4. Collaboration and contribution to the research base, economy and society

This section should provide information about the submitted unit’s research

collaborations, networks and partnerships, including relationships with key research users,

beneficiaries or audiences; and the wider activities and contributions to the research base,

economy and society.

REF 2021

101

Annex J: Standard data analyses

When we provide submissions to sub-panels, we will supply a standard analysis of the

data submitted in REF4a/b/c, and some of the data submitted in REF1a/b or obtained

through the HESA staff record, in respect of each submission in that UOA, and aggregated

for all submissions in that UOA (see paragraph 338). The items listed below will be provided

to panels.

Summary of each submission within a UOA

Total number of Category A submitted staff, and ECR status (headcount).

Total number of Category A submitted staff (FTE).

Percentage of Category A submitted staff out of Category A eligible population (FTE).

Total number of outputs submitted.

Numbers and percentage of Category A submitted, ECRs and former staff (headcount)

with 1, 2, 3, 4 or 5 research outputs attributed to them.

Number of research doctoral degrees awarded, by year.

Number of research doctoral degrees awarded per submitted staff

, by year.

Average annual research income, total across all sources.

Average annual research income per submitted staff (see footnote 2121), total across

all sources.

10.

Average annual research income by source.

11.

Average annual research income per submitted staff (see footnote 21) by source.

UOA summary

Total number of Category A submitted staff, and ECR status (headcount).

Total number of Category A submitted staff (FTE).

Average percentage of Category A submitted staff out of Category A eligible

population (FTE).

Total number of outputs submitted.

Average number and percentage of Category A submitted, ECRs and former staff

(headcount) with 1, 2, 3, 4 or 5 research outputs attributed to them.

Average number of research doctoral degrees awarded, by year.

Average number of research doctoral degrees awarded per submitted staff

(see footnote 21), by year.

Average annual research income, total across all sources.

Average and median annual research income per submitted staff (see footnote 21),

total across all sources.

10.

Average annual research income by source.

11.

Average and median annual research income per submitted staff (see footnote 21)

by source.

21.

Two versions of this indicator will be shown: one using headcount of Category A submitted staff as the

denominator; the second using FTE of Category A submitted staff as the denominator.

102

REF 2021

Annex K: Output glossary and collection formats for REF2

and REF3

An underpinning principle of the REF is that all forms of research output will be assessed

on a fair and equal basis. Sub-panels will not regard any particular form of output as of

greater or lesser quality than another per se.

All research outputs must meet the definition of research for the REF. For the purposes

of the REF, research is defined as ‘a process of investigation leading to new insights,

effectively shared’. Please see Annex C for further detail.

All submissions should provide sufficient information to allow a sub-panel to

understand the research process, the research insights, and the time and manner of

dissemination. Often this will be evident within the materials submitted, requiring no

further information. However, where this is not evident within the submission, this may be

supplemented by an up to 300-word statement or supporting evidence. See ‘Panel criteria’,

Annex B, for a summary of the additional information requirements for outputs.

The table below sets out categories of output types under which outputs will be

submitted in REF 2021, the collection formats for the different output types, and a broad

definition of each category. This includes examples, which are provided for guidance only

and do not represent a definitive list.